{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "876db82e",
   "metadata": {},
   "source": [
    "# Prot T5 Finetuning\n",
    "# per residue regression\n",
    "\n",
    "This notebook allows you to finetune ProtT5 to your own datasets\n",
    "\n",
    "The protein language model ProtT5 was first published [here](https://ieeexplore.ieee.org/document/9477085) and is available on [github](https://github.com/agemagician/ProtTrans). We use the [huggingface](https://huggingface.co/Rostlab/prot_t5_xl_uniref50) checkpoint.\n",
    "\n",
    "For better perfomance we apply [Parameter-Efficient Fine-Tuning (PEFT)](https://huggingface.co/blog/peft). For this we apply [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685).\n",
    "\n",
    "For higher memory efficiency we also utilize the [deepspeed](https://github.com/microsoft/DeepSpeed) implementation of [huggingface](https://huggingface.co/docs/accelerate/usage_guides/deepspeed).\n",
    "\n",
    "The core training loop is implemented with the pytorch [huggingface trainer](https://huggingface.co/docs/transformers/main_classes/trainer) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5a5ec35",
   "metadata": {},
   "source": [
    "## Imports and env. variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "angry-toronto",
   "metadata": {},
   "outputs": [],
   "source": [
    "#import dependencies\n",
    "import os.path\n",
    "os.chdir(\"set a path here\")\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "import re\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import copy\n",
    "\n",
    "import transformers, datasets\n",
    "from transformers.modeling_outputs import TokenClassifierOutput\n",
    "from transformers.models.t5.modeling_t5 import T5Config, T5PreTrainedModel, T5Stack\n",
    "from transformers.utils.model_parallel_utils import assert_device_map, get_device_map\n",
    "from transformers import T5EncoderModel, T5Tokenizer\n",
    "from transformers import TrainingArguments, Trainer, set_seed\n",
    "\n",
    "#DataCollator\n",
    "from transformers.data.data_collator import DataCollatorMixin\n",
    "from transformers.tokenization_utils_base import PreTrainedTokenizerBase\n",
    "from transformers.utils import PaddingStrategy\n",
    "\n",
    "import random\n",
    "import warnings\n",
    "from collections.abc import Mapping\n",
    "from dataclasses import dataclass\n",
    "from random import randint\n",
    "from typing import Any, Callable, Dict, List, NewType, Optional, Tuple, Union\n",
    "\n",
    "from evaluate import load\n",
    "from datasets import Dataset\n",
    "\n",
    "from tqdm import tqdm\n",
    "import random\n",
    "\n",
    "from scipy import stats\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8534fbd4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set environment variables to run Deepspeed from a notebook\n",
    "os.environ[\"MASTER_ADDR\"] = \"localhost\"\n",
    "os.environ[\"MASTER_PORT\"] = \"9993\"  # modify if RuntimeError: Address already in use\n",
    "os.environ[\"RANK\"] = \"0\"\n",
    "os.environ[\"LOCAL_RANK\"] = \"0\"\n",
    "os.environ[\"WORLD_SIZE\"] = \"1\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "808bb08d",
   "metadata": {},
   "source": [
    "# Environment to run this notebook\n",
    "\n",
    "\n",
    "These are the versions of the core packages we use to run this notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b35bdadd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Torch version:  1.13.1\n",
      "Cuda version:  11.7\n",
      "Numpy version:  1.22.3\n",
      "Pandas version:  1.5.3\n",
      "Transformers version:  4.26.1\n",
      "Datasets version:  2.9.0\n"
     ]
    }
   ],
   "source": [
    "print(\"Torch version: \",torch.__version__)\n",
    "print(\"Cuda version: \",torch.version.cuda)\n",
    "print(\"Numpy version: \",np.__version__)\n",
    "print(\"Pandas version: \",pd.__version__)\n",
    "print(\"Transformers version: \",transformers.__version__)\n",
    "print(\"Datasets version: \",datasets.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d109a4d7",
   "metadata": {},
   "source": [
    "**For easy setup of this environment you can use the finetuning.yml File provided in this folder**\n",
    "\n",
    "check here for [setting up env from a yml File](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb2dda19",
   "metadata": {},
   "source": [
    "# Input data\n",
    "\n",
    "Provide your training and validation data in seperate pandas dataframes \n",
    "\n",
    "example shown below"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c012178",
   "metadata": {},
   "source": [
    "**Modify the data loading part above as needed for your data**\n",
    "\n",
    "To run the training you need two dataframes (training and validation) each with the columns \"sequence\" and \"label\"\n",
    "\n",
    "Columns are:\n",
    "+ protein sequence\n",
    "+ label is a list of len(protein sequence) with float numbers corresponding to predicted regression value at this position\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6a10d4e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# For this example we import the disorder dataset from https://github.com/DagmarIlz/SETH/\n",
    "# For details, see publication here: https://www.frontiersin.org/articles/10.3389/fbinf.2022.1019597/full\n",
    "\n",
    "from Bio import SeqIO\n",
    "from io import StringIO\n",
    "import requests\n",
    "import tempfile\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "import csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a84e2021",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>sequence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>26672</td>\n",
       "      <td>MASNDYTQQATQSYGAYPTQPGQGYSQQSSQPYGQQSYSGYSQSTD...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>26654</td>\n",
       "      <td>GTRGDADMYDLPKKEDALLYQSKGYNDDYYEESYFTTRTYGEPESA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>25586</td>\n",
       "      <td>SLTLNLITEMGRLPTFMTQKARDALDNLAVLHTAEAGGRAYNHALSEL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>25399</td>\n",
       "      <td>RLDKQGNFNAWVAGSYGNDQWLQVDLGSSKEVTGIITQGARNFGSV...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>25185</td>\n",
       "      <td>MGTSAPNNTNNANSSITPAFGSNNTGNTAFGNSNPTSNVFGSNNST...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    name                                           sequence\n",
       "0  26672  MASNDYTQQATQSYGAYPTQPGQGYSQQSSQPYGQQSYSGYSQSTD...\n",
       "1  26654  GTRGDADMYDLPKKEDALLYQSKGYNDDYYEESYFTTRTYGEPESA...\n",
       "2  25586   SLTLNLITEMGRLPTFMTQKARDALDNLAVLHTAEAGGRAYNHALSEL\n",
       "3  25399  RLDKQGNFNAWVAGSYGNDQWLQVDLGSSKEVTGIITQGARNFGSV...\n",
       "4  25185  MGTSAPNNTNNANSSITPAFGSNNTGNTAFGNSNPTSNVFGSNNST..."
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load training sequences\n",
    "url = \"https://raw.githubusercontent.com/DagmarIlz/SETH/main/datasets/CheZOD1174_training_set_sequences.fasta\"\n",
    "\n",
    "response = requests.get(url)\n",
    "response.raise_for_status()  # Check if the request was successful\n",
    "\n",
    "# Create a StringIO object to simulate a file-like object\n",
    "fasta_file = StringIO(response.text)\n",
    "\n",
    "# Load FASTA file using Biopython\n",
    "sequences = []\n",
    "for record in SeqIO.parse(fasta_file, \"fasta\"):\n",
    "    sequences.append([record.name, str(record.seq)])\n",
    "\n",
    "# Create dataframe\n",
    "df = pd.DataFrame(sequences, columns=[\"name\", \"sequence\"])\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2e0c81ba",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "# Load training labels (CheZOD_scores)\n",
    "url = \"https://raw.githubusercontent.com/DagmarIlz/SETH/main/datasets/CheZOD1174_training_set_CheZOD_scores.txt\"\n",
    "\n",
    "response = requests.get(url)\n",
    "response.raise_for_status()  # Check if the request was successful\n",
    "\n",
    "lines = response.text.splitlines()\n",
    "\n",
    "names=[]\n",
    "labels=[]\n",
    "\n",
    "# Split each line into name and label\n",
    "for l in lines:\n",
    "    names.append(l.split(\":\\t\")[0])\n",
    "    labels.append(l.split(\":\\t\")[1].split(\", \"))\n",
    "\n",
    "# Covert labels to float values\n",
    "for l in range(0,len(labels)):\n",
    "    labels[l]=[float(label) for label in labels[l]]\n",
    "    \n",
    "# check if sequence names match\n",
    "print(list(df.name) == names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7921649f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sequence</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>GVGETKVIYHLDEEETPYLVKIPVPAERITLGDFKSVLQRPAGAKY...</td>\n",
       "      <td>[-100, 3.219, 4.014, 3.745, 3.53, 2.301, 3.204...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>GSSGSSGRKKPVSQSLEFPTRYSPYRPYRCVHQGCFAAFTIQQNLI...</td>\n",
       "      <td>[-100, -100, -100, -100, -100, -100, -2.194, -...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>GIHKQKEKSRLQGGVLVNEILNHMKRATQIPSYKKLIMY</td>\n",
       "      <td>[-100, 6.476, 6.296, 7.407, 5.346, 5.509, 4.99...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>MGMIVFVRFNSSYGFPVEVDSDTSILQLKEVVAKRQGVPADQLRVI...</td>\n",
       "      <td>[-100, 9.189, 10.389, 12.882, 12.002, 14.043, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>SVLQVLHIPDERLRKVAKPVEEVNAEIQRIVDDMFETMYAEEGIGL...</td>\n",
       "      <td>[-100, 9.978, 12.505, 13.381, 13.069, 13.551, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1051</th>\n",
       "      <td>MHHHHHHGSTKTNSEILEQLKQASDGLLFMSESEYPFEVFLWEGSA...</td>\n",
       "      <td>[-100, -100, -100, -100, -100, -100, -100, -0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1052</th>\n",
       "      <td>MAEALFKEIDVNGDGAVSYEEVKAFVSKKRAIKNEQLLQLIFKSID...</td>\n",
       "      <td>[-100, -100, -100, 11.302, 13.574, 14.298, 14....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1053</th>\n",
       "      <td>MSNLEIKQGENKFYIGDDENNALAEITYRFVDNNEINIDHTGVSDE...</td>\n",
       "      <td>[-100, 4.412, 8.79, 10.449, 12.936, 13.87, 13....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1054</th>\n",
       "      <td>SMTKDNEVEQEDLAQSLSLVKDVIGAVDSKVASYEKKVRLNEIYTK...</td>\n",
       "      <td>[-100, 0.479, 0.133, 0.265, -1.242, -0.704, -0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1055</th>\n",
       "      <td>AEGDDPAKAAFNSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFT...</td>\n",
       "      <td>[-100, 5.33, 6.212, 8.44, 8.814, 11.532, 13.12...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1056 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               sequence  \\\n",
       "0     GVGETKVIYHLDEEETPYLVKIPVPAERITLGDFKSVLQRPAGAKY...   \n",
       "1     GSSGSSGRKKPVSQSLEFPTRYSPYRPYRCVHQGCFAAFTIQQNLI...   \n",
       "2               GIHKQKEKSRLQGGVLVNEILNHMKRATQIPSYKKLIMY   \n",
       "3     MGMIVFVRFNSSYGFPVEVDSDTSILQLKEVVAKRQGVPADQLRVI...   \n",
       "4     SVLQVLHIPDERLRKVAKPVEEVNAEIQRIVDDMFETMYAEEGIGL...   \n",
       "...                                                 ...   \n",
       "1051  MHHHHHHGSTKTNSEILEQLKQASDGLLFMSESEYPFEVFLWEGSA...   \n",
       "1052  MAEALFKEIDVNGDGAVSYEEVKAFVSKKRAIKNEQLLQLIFKSID...   \n",
       "1053  MSNLEIKQGENKFYIGDDENNALAEITYRFVDNNEINIDHTGVSDE...   \n",
       "1054  SMTKDNEVEQEDLAQSLSLVKDVIGAVDSKVASYEKKVRLNEIYTK...   \n",
       "1055  AEGDDPAKAAFNSLQASATEYIGYAWAMVVVIVGATIGIKLFKKFT...   \n",
       "\n",
       "                                                  label  \n",
       "0     [-100, 3.219, 4.014, 3.745, 3.53, 2.301, 3.204...  \n",
       "1     [-100, -100, -100, -100, -100, -100, -2.194, -...  \n",
       "2     [-100, 6.476, 6.296, 7.407, 5.346, 5.509, 4.99...  \n",
       "3     [-100, 9.189, 10.389, 12.882, 12.002, 14.043, ...  \n",
       "4     [-100, 9.978, 12.505, 13.381, 13.069, 13.551, ...  \n",
       "...                                                 ...  \n",
       "1051  [-100, -100, -100, -100, -100, -100, -100, -0....  \n",
       "1052  [-100, -100, -100, 11.302, 13.574, 14.298, 14....  \n",
       "1053  [-100, 4.412, 8.79, 10.449, 12.936, 13.87, 13....  \n",
       "1054  [-100, 0.479, 0.133, 0.265, -1.242, -0.704, -0...  \n",
       "1055  [-100, 5.33, 6.212, 8.44, 8.814, 11.532, 13.12...  \n",
       "\n",
       "[1056 rows x 2 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create train and validation dataframes\n",
    "\n",
    "# Add label column\n",
    "df[\"label\"] = labels\n",
    "# drop name column\n",
    "df = df [[\"sequence\", \"label\"]]\n",
    "\n",
    "# Split of 10% of training data as validation split\n",
    "train, valid = train_test_split(df, test_size=0.1, random_state=42)\n",
    "train.reset_index(drop=True,inplace=True)\n",
    "valid.reset_index(drop=True,inplace=True)\n",
    "\n",
    "# Replace invalid labels (>900) with -100 (will be ignored by pytorch loss)\n",
    "train['label'] = train.apply(lambda row:  [-100 if x > 900 else x for x in row['label']], axis=1)\n",
    "valid['label'] = valid.apply(lambda row:  [-100 if x > 900 else x for x in row['label']], axis=1)\n",
    "\n",
    "train"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b996723",
   "metadata": {},
   "source": [
    "# PT5 Model and Low Rank Adaptation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "responsible-standing",
   "metadata": {},
   "source": [
    "## LoRA modification definition\n",
    "\n",
    "Implementation taken from https://github.com/r-three/t-few\n",
    "\n",
    "(https://github.com/r-three/t-few/blob/master/src/models/lora.py, https://github.com/r-three/t-few/tree/master/configs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "separated-grenada",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Modifies an existing transformer and introduce the LoRA layers\n",
    "\n",
    "class LoRAConfig:\n",
    "    def __init__(self):\n",
    "        self.lora_rank = 4\n",
    "        self.lora_init_scale = 0.01\n",
    "        self.lora_modules = \".*SelfAttention|.*EncDecAttention\"\n",
    "        self.lora_layers = \"q|k|v|o\"\n",
    "        self.trainable_param_names = \".*layer_norm.*|.*lora_[ab].*\"\n",
    "        self.lora_scaling_rank = 1\n",
    "        # lora_modules and lora_layers are speicified with regular expressions\n",
    "        # see https://www.w3schools.com/python/python_regex.asp for reference\n",
    "        \n",
    "class LoRALinear(nn.Module):\n",
    "    def __init__(self, linear_layer, rank, scaling_rank, init_scale):\n",
    "        super().__init__()\n",
    "        self.in_features = linear_layer.in_features\n",
    "        self.out_features = linear_layer.out_features\n",
    "        self.rank = rank\n",
    "        self.scaling_rank = scaling_rank\n",
    "        self.weight = linear_layer.weight\n",
    "        self.bias = linear_layer.bias\n",
    "        if self.rank > 0:\n",
    "            self.lora_a = nn.Parameter(torch.randn(rank, linear_layer.in_features) * init_scale)\n",
    "            if init_scale < 0:\n",
    "                self.lora_b = nn.Parameter(torch.randn(linear_layer.out_features, rank) * init_scale)\n",
    "            else:\n",
    "                self.lora_b = nn.Parameter(torch.zeros(linear_layer.out_features, rank))\n",
    "        if self.scaling_rank:\n",
    "            self.multi_lora_a = nn.Parameter(\n",
    "                torch.ones(self.scaling_rank, linear_layer.in_features)\n",
    "                + torch.randn(self.scaling_rank, linear_layer.in_features) * init_scale\n",
    "            )\n",
    "            if init_scale < 0:\n",
    "                self.multi_lora_b = nn.Parameter(\n",
    "                    torch.ones(linear_layer.out_features, self.scaling_rank)\n",
    "                    + torch.randn(linear_layer.out_features, self.scaling_rank) * init_scale\n",
    "                )\n",
    "            else:\n",
    "                self.multi_lora_b = nn.Parameter(torch.ones(linear_layer.out_features, self.scaling_rank))\n",
    "\n",
    "    def forward(self, input):\n",
    "        if self.scaling_rank == 1 and self.rank == 0:\n",
    "            # parsimonious implementation for ia3 and lora scaling\n",
    "            if self.multi_lora_a.requires_grad:\n",
    "                hidden = F.linear((input * self.multi_lora_a.flatten()), self.weight, self.bias)\n",
    "            else:\n",
    "                hidden = F.linear(input, self.weight, self.bias)\n",
    "            if self.multi_lora_b.requires_grad:\n",
    "                hidden = hidden * self.multi_lora_b.flatten()\n",
    "            return hidden\n",
    "        else:\n",
    "            # general implementation for lora (adding and scaling)\n",
    "            weight = self.weight\n",
    "            if self.scaling_rank:\n",
    "                weight = weight * torch.matmul(self.multi_lora_b, self.multi_lora_a) / self.scaling_rank\n",
    "            if self.rank:\n",
    "                weight = weight + torch.matmul(self.lora_b, self.lora_a) / self.rank\n",
    "            return F.linear(input, weight, self.bias)\n",
    "\n",
    "    def extra_repr(self):\n",
    "        return \"in_features={}, out_features={}, bias={}, rank={}, scaling_rank={}\".format(\n",
    "            self.in_features, self.out_features, self.bias is not None, self.rank, self.scaling_rank\n",
    "        )\n",
    "\n",
    "\n",
    "def modify_with_lora(transformer, config):\n",
    "    for m_name, module in dict(transformer.named_modules()).items():\n",
    "        if re.fullmatch(config.lora_modules, m_name):\n",
    "            for c_name, layer in dict(module.named_children()).items():\n",
    "                if re.fullmatch(config.lora_layers, c_name):\n",
    "                    assert isinstance(\n",
    "                        layer, nn.Linear\n",
    "                    ), f\"LoRA can only be applied to torch.nn.Linear, but {layer} is {type(layer)}.\"\n",
    "                    setattr(\n",
    "                        module,\n",
    "                        c_name,\n",
    "                        LoRALinear(layer, config.lora_rank, config.lora_scaling_rank, config.lora_init_scale),\n",
    "                    )\n",
    "    return transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "electronic-channels",
   "metadata": {},
   "source": [
    "## Classification model definition \n",
    "\n",
    "adding a token classification head on top of the encoder model\n",
    "\n",
    "modified from [EsmForTokenClassification](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/esm/modeling_esm.py#L1178)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "acting-archives",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ClassConfig:\n",
    "    def __init__(self, dropout=0.2, num_labels=1):\n",
    "        self.dropout_rate = dropout\n",
    "        self.num_labels = num_labels\n",
    "\n",
    "class T5EncoderForTokenClassification(T5PreTrainedModel):\n",
    "\n",
    "    def __init__(self, config: T5Config, class_config):\n",
    "        super().__init__(config)\n",
    "        self.num_labels = class_config.num_labels\n",
    "        self.config = config\n",
    "\n",
    "        self.shared = nn.Embedding(config.vocab_size, config.d_model)\n",
    "\n",
    "        encoder_config = copy.deepcopy(config)\n",
    "        encoder_config.use_cache = False\n",
    "        encoder_config.is_encoder_decoder = False\n",
    "        self.encoder = T5Stack(encoder_config, self.shared)\n",
    "\n",
    "        self.dropout = nn.Dropout(class_config.dropout_rate) \n",
    "        self.classifier = nn.Linear(config.hidden_size, class_config.num_labels)\n",
    "\n",
    "        # Initialize weights and apply final processing\n",
    "        self.post_init()\n",
    "\n",
    "        # Model parallel\n",
    "        self.model_parallel = False\n",
    "        self.device_map = None\n",
    "\n",
    "    def parallelize(self, device_map=None):\n",
    "        self.device_map = (\n",
    "            get_device_map(len(self.encoder.block), range(torch.cuda.device_count()))\n",
    "            if device_map is None\n",
    "            else device_map\n",
    "        )\n",
    "        assert_device_map(self.device_map, len(self.encoder.block))\n",
    "        self.encoder.parallelize(self.device_map)\n",
    "        self.classifier = self.classifier.to(self.encoder.first_device)\n",
    "        self.model_parallel = True\n",
    "\n",
    "    def deparallelize(self):\n",
    "        self.encoder.deparallelize()\n",
    "        self.encoder = self.encoder.to(\"cpu\")\n",
    "        self.model_parallel = False\n",
    "        self.device_map = None\n",
    "        torch.cuda.empty_cache()\n",
    "\n",
    "    def get_input_embeddings(self):\n",
    "        return self.shared\n",
    "\n",
    "    def set_input_embeddings(self, new_embeddings):\n",
    "        self.shared = new_embeddings\n",
    "        self.encoder.set_input_embeddings(new_embeddings)\n",
    "\n",
    "    def get_encoder(self):\n",
    "        return self.encoder\n",
    "\n",
    "    def _prune_heads(self, heads_to_prune):\n",
    "        \"\"\"\n",
    "        Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base\n",
    "        class PreTrainedModel\n",
    "        \"\"\"\n",
    "        for layer, heads in heads_to_prune.items():\n",
    "            self.encoder.layer[layer].attention.prune_heads(heads)\n",
    "\n",
    "    def forward(\n",
    "        self,\n",
    "        input_ids=None,\n",
    "        attention_mask=None,\n",
    "        head_mask=None,\n",
    "        inputs_embeds=None,\n",
    "        labels=None,\n",
    "        output_attentions=None,\n",
    "        output_hidden_states=None,\n",
    "        return_dict=None,\n",
    "    ):\n",
    "        return_dict = return_dict if return_dict is not None else self.config.use_return_dict\n",
    "\n",
    "        outputs = self.encoder(\n",
    "            input_ids=input_ids,\n",
    "            attention_mask=attention_mask,\n",
    "            inputs_embeds=inputs_embeds,\n",
    "            head_mask=head_mask,\n",
    "            output_attentions=output_attentions,\n",
    "            output_hidden_states=output_hidden_states,\n",
    "            return_dict=return_dict,\n",
    "        )\n",
    "\n",
    "        sequence_output = outputs[0]\n",
    "        sequence_output = self.dropout(sequence_output)\n",
    "        logits = self.classifier(sequence_output)\n",
    "\n",
    "        loss = None\n",
    "        if labels is not None:\n",
    "            loss_fct = MSELoss()\n",
    "\n",
    "            active_loss = attention_mask.view(-1) == 1\n",
    "            active_logits = logits.view(-1)\n",
    "\n",
    "            active_labels = torch.where(\n",
    "              active_loss, labels.view(-1), torch.tensor(-100).type_as(labels)\n",
    "            )\n",
    "\n",
    "            valid_logits=active_logits[active_labels!=-100]\n",
    "            valid_labels=active_labels[active_labels!=-100]\n",
    "\n",
    "            loss = loss_fct(valid_logits, valid_labels)\n",
    "            \n",
    "        \n",
    "        if not return_dict:\n",
    "            output = (logits,) + outputs[2:]\n",
    "            return ((loss,) + output) if loss is not None else output\n",
    "\n",
    "        return TokenClassifierOutput(\n",
    "            loss=loss,\n",
    "            logits=logits,\n",
    "            hidden_states=outputs.hidden_states,\n",
    "            attentions=outputs.attentions,\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71a0e217",
   "metadata": {},
   "source": [
    "## Modified ProtT5 model\n",
    "this creates a ProtT5 model with prediction head and LoRA modification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "split-austin",
   "metadata": {},
   "outputs": [],
   "source": [
    "def PT5_classification_model(num_labels):\n",
    "    # Load PT5 and tokenizer\n",
    "    model = T5EncoderModel.from_pretrained(\"Rostlab/prot_t5_xl_uniref50\")\n",
    "    tokenizer = T5Tokenizer.from_pretrained(\"Rostlab/prot_t5_xl_uniref50\") \n",
    "    \n",
    "    # Create new Classifier model with PT5 dimensions\n",
    "    class_config=ClassConfig(num_labels=num_labels)\n",
    "    class_model=T5EncoderForTokenClassification(model.config,class_config)\n",
    "    \n",
    "    # Set encoder and embedding weights to checkpoint weights\n",
    "    class_model.shared=model.shared\n",
    "    class_model.encoder=model.encoder    \n",
    "    \n",
    "    # Delete the checkpoint model\n",
    "    model=class_model\n",
    "    del class_model\n",
    "    \n",
    "    # Print number of trainable parameters\n",
    "    model_parameters = filter(lambda p: p.requires_grad, model.parameters())\n",
    "    params = sum([np.prod(p.size()) for p in model_parameters])\n",
    "    print(\"ProtT5_Classfier\\nTrainable Parameter: \"+ str(params))    \n",
    " \n",
    "    # Add model modification lora\n",
    "    config = LoRAConfig()\n",
    "    \n",
    "    # Add LoRA layers\n",
    "    model = modify_with_lora(model, config)\n",
    "    \n",
    "    # Freeze Embeddings and Encoder (except LoRA)\n",
    "    for (param_name, param) in model.shared.named_parameters():\n",
    "                param.requires_grad = False\n",
    "    for (param_name, param) in model.encoder.named_parameters():\n",
    "                param.requires_grad = False       \n",
    "\n",
    "    for (param_name, param) in model.named_parameters():\n",
    "            if re.fullmatch(config.trainable_param_names, param_name):\n",
    "                param.requires_grad = True\n",
    "\n",
    "    # Print trainable Parameter          \n",
    "    model_parameters = filter(lambda p: p.requires_grad, model.parameters())\n",
    "    params = sum([np.prod(p.size()) for p in model_parameters])\n",
    "    print(\"ProtT5_LoRA_Classfier\\nTrainable Parameter: \"+ str(params) + \"\\n\")\n",
    "    \n",
    "    return model, tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beautiful-yeast",
   "metadata": {},
   "source": [
    "# Training Definition "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e735e819",
   "metadata": {},
   "source": [
    "## Deepspeed config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "eed91c2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Deepspeed config for optimizer CPU offload\n",
    "\n",
    "ds_config = {\n",
    "    \"fp16\": {\n",
    "        \"enabled\": \"auto\",\n",
    "        \"loss_scale\": 0,\n",
    "        \"loss_scale_window\": 1000,\n",
    "        \"initial_scale_power\": 16,\n",
    "        \"hysteresis\": 2,\n",
    "        \"min_loss_scale\": 1\n",
    "    },\n",
    "\n",
    "    \"optimizer\": {\n",
    "        \"type\": \"AdamW\",\n",
    "        \"params\": {\n",
    "            \"lr\": \"auto\",\n",
    "            \"betas\": \"auto\",\n",
    "            \"eps\": \"auto\",\n",
    "            \"weight_decay\": \"auto\"\n",
    "        }\n",
    "    },\n",
    "\n",
    "    \"scheduler\": {\n",
    "        \"type\": \"WarmupLR\",\n",
    "        \"params\": {\n",
    "            \"warmup_min_lr\": \"auto\",\n",
    "            \"warmup_max_lr\": \"auto\",\n",
    "            \"warmup_num_steps\": \"auto\"\n",
    "        }\n",
    "    },\n",
    "\n",
    "    \"zero_optimization\": {\n",
    "        \"stage\": 2,\n",
    "        \"offload_optimizer\": {\n",
    "            \"device\": \"cpu\",\n",
    "            \"pin_memory\": True\n",
    "        },\n",
    "        \"allgather_partitions\": True,\n",
    "        \"allgather_bucket_size\": 2e8,\n",
    "        \"overlap_comm\": True,\n",
    "        \"reduce_scatter\": True,\n",
    "        \"reduce_bucket_size\": 2e8,\n",
    "        \"contiguous_gradients\": True\n",
    "    },\n",
    "\n",
    "    \"gradient_accumulation_steps\": \"auto\",\n",
    "    \"gradient_clipping\": \"auto\",\n",
    "    \"steps_per_print\": 2000,\n",
    "    \"train_batch_size\": \"auto\",\n",
    "    \"train_micro_batch_size_per_gpu\": \"auto\",\n",
    "    \"wall_clock_breakdown\": False\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b10da92",
   "metadata": {},
   "source": [
    "## Data Collator (using float instead of ints as labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "416686f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "@dataclass\n",
    "class DataCollatorForTokenRegression(DataCollatorMixin):\n",
    "    \"\"\"\n",
    "    Data collator that will dynamically pad the inputs received, as well as the labels.\n",
    "    Args:\n",
    "        tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]):\n",
    "            The tokenizer used for encoding the data.\n",
    "        padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`):\n",
    "            Select a strategy to pad the returned sequences (according to the model's padding side and padding index)\n",
    "            among:\n",
    "            - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single\n",
    "              sequence is provided).\n",
    "            - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum\n",
    "              acceptable input length for the model if that argument is not provided.\n",
    "            - `False` or `'do_not_pad'`: No padding (i.e., can output a batch with sequences of different lengths).\n",
    "        max_length (`int`, *optional*):\n",
    "            Maximum length of the returned list and optionally padding length (see above).\n",
    "        pad_to_multiple_of (`int`, *optional*):\n",
    "            If set will pad the sequence to a multiple of the provided value.\n",
    "            This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >=\n",
    "            7.5 (Volta).\n",
    "        label_pad_token_id (`int`, *optional*, defaults to -100):\n",
    "            The id to use when padding the labels (-100 will be automatically ignore by PyTorch loss functions).\n",
    "        return_tensors (`str`):\n",
    "            The type of Tensor to return. Allowable values are \"np\", \"pt\" and \"tf\".\n",
    "    \"\"\"\n",
    "\n",
    "    tokenizer: PreTrainedTokenizerBase\n",
    "    padding: Union[bool, str, PaddingStrategy] = True\n",
    "    max_length: Optional[int] = None\n",
    "    pad_to_multiple_of: Optional[int] = None\n",
    "    label_pad_token_id: int = -100\n",
    "    return_tensors: str = \"pt\"\n",
    "\n",
    "    def torch_call(self, features):\n",
    "        import torch\n",
    "\n",
    "        label_name = \"label\" if \"label\" in features[0].keys() else \"labels\"\n",
    "        labels = [feature[label_name] for feature in features] if label_name in features[0].keys() else None\n",
    "\n",
    "        no_labels_features = [{k: v for k, v in feature.items() if k != label_name} for feature in features]\n",
    "\n",
    "        batch = self.tokenizer.pad(\n",
    "            no_labels_features,\n",
    "            padding=self.padding,\n",
    "            max_length=self.max_length,\n",
    "            pad_to_multiple_of=self.pad_to_multiple_of,\n",
    "            return_tensors=\"pt\",\n",
    "        )\n",
    "\n",
    "        if labels is None:\n",
    "            return batch\n",
    "\n",
    "        sequence_length = batch[\"input_ids\"].shape[1]\n",
    "        padding_side = self.tokenizer.padding_side\n",
    "\n",
    "        def to_list(tensor_or_iterable):\n",
    "            if isinstance(tensor_or_iterable, torch.Tensor):\n",
    "                return tensor_or_iterable.tolist()\n",
    "            return list(tensor_or_iterable)\n",
    "\n",
    "        if padding_side == \"right\":\n",
    "            batch[label_name] = [\n",
    "                to_list(label) + [self.label_pad_token_id] * (sequence_length - len(label)) for label in labels\n",
    "\n",
    "            ]\n",
    "        else:\n",
    "            batch[label_name] = [\n",
    "                [self.label_pad_token_id] * (sequence_length - len(label)) + to_list(label) for label in labels\n",
    "            ]\n",
    "\n",
    "        batch[label_name] = torch.tensor(batch[label_name], dtype=torch.float)\n",
    "        return batch\n",
    "\n",
    "def _torch_collate_batch(examples, tokenizer, pad_to_multiple_of: Optional[int] = None):\n",
    "    \"\"\"Collate `examples` into a batch, using the information in `tokenizer` for padding if necessary.\"\"\"\n",
    "    import torch\n",
    "\n",
    "    # Tensorize if necessary.\n",
    "    if isinstance(examples[0], (list, tuple, np.ndarray)):\n",
    "        examples = [torch.tensor(e, dtype=torch.long) for e in examples]\n",
    "\n",
    "    length_of_first = examples[0].size(0)\n",
    "\n",
    "    # Check if padding is necessary.\n",
    "\n",
    "    are_tensors_same_length = all(x.size(0) == length_of_first for x in examples)\n",
    "    if are_tensors_same_length and (pad_to_multiple_of is None or length_of_first % pad_to_multiple_of == 0):\n",
    "        return torch.stack(examples, dim=0)\n",
    "\n",
    "    # If yes, check if we have a `pad_token`.\n",
    "    if tokenizer._pad_token is None:\n",
    "        raise ValueError(\n",
    "            \"You are attempting to pad samples but the tokenizer you are using\"\n",
    "            f\" ({tokenizer.__class__.__name__}) does not have a pad token.\"\n",
    "        )\n",
    "\n",
    "    # Creating the full tensor and filling it with our data.\n",
    "    max_length = max(x.size(0) for x in examples)\n",
    "    if pad_to_multiple_of is not None and (max_length % pad_to_multiple_of != 0):\n",
    "        max_length = ((max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of\n",
    "    result = examples[0].new_full([len(examples), max_length], tokenizer.pad_token_id)\n",
    "    for i, example in enumerate(examples):\n",
    "        if tokenizer.padding_side == \"right\":\n",
    "            result[i, : example.shape[0]] = example\n",
    "        else:\n",
    "            result[i, -example.shape[0] :] = example\n",
    "    return result\n",
    "\n",
    "def tolist(x):\n",
    "    if isinstance(x, list):\n",
    "        return x\n",
    "    elif hasattr(x, \"numpy\"):  # Checks for TF tensors without needing the import\n",
    "        x = x.numpy()\n",
    "    return x.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92962861",
   "metadata": {},
   "source": [
    "## Training functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "liberal-learning",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set random seeds for reproducibility of your trainings run\n",
    "def set_seeds(s):\n",
    "    torch.manual_seed(s)\n",
    "    np.random.seed(s)\n",
    "    random.seed(s)\n",
    "    set_seed(s)\n",
    "\n",
    "# Dataset creation\n",
    "def create_dataset(tokenizer,seqs,labels):\n",
    "    tokenized = tokenizer(seqs, max_length=1024, padding=True, truncation=True)\n",
    "    dataset = Dataset.from_dict(tokenized)\n",
    "    # we need to cut of labels after 1023 positions for the data collator to add the correct padding (1023 + 1 special tokens)\n",
    "    labels = [l[:1023] for l in labels] \n",
    "    dataset = dataset.add_column(\"labels\", labels)\n",
    "     \n",
    "    return dataset\n",
    "    \n",
    "# Main training fuction\n",
    "def train_per_residue(\n",
    "        train_df,         #training data\n",
    "        valid_df,         #validation data      \n",
    "        num_labels= 1,    #number of classes\n",
    "    \n",
    "        # effective training batch size is batch * accum\n",
    "        # we recommend an effective batch size of 8 \n",
    "        batch= 4,         #for training\n",
    "        accum= 2,         #gradient accumulation\n",
    "    \n",
    "        val_batch = 16,   #batch size for evaluation\n",
    "        epochs= 10,       #training epochs\n",
    "        lr= 3e-4,         #recommended learning rate\n",
    "        seed= 42,         #random seed\n",
    "        deepspeed= True,  #if gpu is large enough disable deepspeed for training speedup\n",
    "        gpu= 1 ):         #gpu selection (1 for first gpu)\n",
    "\n",
    "    # Set gpu device\n",
    "    os.environ[\"CUDA_VISIBLE_DEVICES\"]=str(gpu-1)\n",
    "    \n",
    "    # Set all random seeds\n",
    "    set_seeds(seed)\n",
    "    \n",
    "    # load model\n",
    "    model, tokenizer = PT5_classification_model(num_labels=num_labels)\n",
    "\n",
    "    # Preprocess inputs\n",
    "    # Replace uncommon AAs with \"X\"\n",
    "    train_df[\"sequence\"]=train_df[\"sequence\"].str.replace('|'.join([\"O\",\"B\",\"U\",\"Z\"]),\"X\",regex=True)\n",
    "    valid_df[\"sequence\"]=valid_df[\"sequence\"].str.replace('|'.join([\"O\",\"B\",\"U\",\"Z\"]),\"X\",regex=True)\n",
    "    # Add spaces between each amino acid for PT5 to correctly use them\n",
    "    train_df['sequence']=train_df.apply(lambda row : \" \".join(row[\"sequence\"]), axis = 1)\n",
    "    valid_df['sequence']=valid_df.apply(lambda row : \" \".join(row[\"sequence\"]), axis = 1)\n",
    "\n",
    "\n",
    "    # Create Datasets\n",
    "    train_set=create_dataset(tokenizer,list(train_df['sequence']),list(train_df['label']))\n",
    "    valid_set=create_dataset(tokenizer,list(valid_df['sequence']),list(valid_df['label']))\n",
    "\n",
    "    # Huggingface Trainer arguments\n",
    "    args = TrainingArguments(\n",
    "        \"./\",\n",
    "        evaluation_strategy = \"steps\",\n",
    "        eval_steps = 528,\n",
    "        logging_strategy = \"epoch\",\n",
    "        save_strategy = \"no\",\n",
    "        learning_rate=lr,\n",
    "        per_device_train_batch_size=batch,\n",
    "        #per_device_eval_batch_size=val_batch,\n",
    "        per_device_eval_batch_size=batch,\n",
    "        gradient_accumulation_steps=accum,\n",
    "        num_train_epochs=epochs,\n",
    "        seed = seed,\n",
    "        deepspeed= ds_config if deepspeed else None,\n",
    "    ) \n",
    "\n",
    "    # Metric definition for validation data\n",
    "    def compute_metrics(eval_pred):\n",
    "\n",
    "        metric = load(\"spearmanr\")\n",
    "        predictions, labels = eval_pred\n",
    "        predictions=predictions.flatten()\n",
    "        labels=labels.flatten()\n",
    "\n",
    "        valid_labels=labels[np.where((labels != -100 ) & (labels < 900 ))]\n",
    "        valid_predictions=predictions[np.where((labels != -100 ) & (labels < 900 ))]\n",
    "\n",
    "        return metric.compute(predictions=valid_predictions, references=valid_labels)\n",
    "\n",
    "    # For token classification we need a data collator here to pad correctly\n",
    "    data_collator = DataCollatorForTokenRegression(tokenizer) \n",
    "\n",
    "    # Trainer          \n",
    "    trainer = Trainer(\n",
    "        model,\n",
    "        args,\n",
    "        train_dataset=train_set,\n",
    "        eval_dataset=valid_set,\n",
    "        tokenizer=tokenizer,\n",
    "        data_collator=data_collator,\n",
    "        compute_metrics=compute_metrics\n",
    "    )    \n",
    "    \n",
    "    # Train model\n",
    "    trainer.train()\n",
    "\n",
    "    return tokenizer, model, trainer.state.log_history\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ac94ab1",
   "metadata": {},
   "source": [
    "# Run Training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ede09d5",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "83fd6b5a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at Rostlab/prot_t5_xl_uniref50 were not used when initializing T5EncoderModel: ['decoder.block.9.layer.1.EncDecAttention.k.weight', 'decoder.block.19.layer.1.EncDecAttention.q.weight', 'decoder.block.10.layer.0.SelfAttention.k.weight', 'decoder.block.9.layer.1.EncDecAttention.v.weight', 'decoder.block.6.layer.0.SelfAttention.k.weight', 'decoder.block.14.layer.0.SelfAttention.k.weight', 'decoder.block.6.layer.0.SelfAttention.o.weight', 'decoder.block.11.layer.2.DenseReluDense.wo.weight', 'decoder.block.13.layer.1.EncDecAttention.v.weight', 'decoder.block.20.layer.1.EncDecAttention.k.weight', 'decoder.block.2.layer.1.EncDecAttention.v.weight', 'decoder.block.0.layer.1.EncDecAttention.v.weight', 'decoder.block.15.layer.0.SelfAttention.o.weight', 'decoder.block.19.layer.0.SelfAttention.k.weight', 'decoder.block.14.layer.2.DenseReluDense.wi.weight', 'decoder.block.22.layer.0.layer_norm.weight', 'decoder.block.5.layer.2.layer_norm.weight', 'decoder.block.1.layer.0.SelfAttention.q.weight', 'decoder.block.11.layer.1.EncDecAttention.q.weight', 'decoder.block.12.layer.0.SelfAttention.o.weight', 'decoder.block.16.layer.1.layer_norm.weight', 'decoder.block.2.layer.1.EncDecAttention.k.weight', 'decoder.block.21.layer.1.EncDecAttention.k.weight', 'decoder.block.23.layer.0.SelfAttention.k.weight', 'decoder.block.9.layer.1.layer_norm.weight', 'decoder.block.3.layer.2.DenseReluDense.wi.weight', 'decoder.embed_tokens.weight', 'decoder.block.21.layer.1.EncDecAttention.v.weight', 'decoder.block.7.layer.1.EncDecAttention.k.weight', 'decoder.block.23.layer.2.DenseReluDense.wo.weight', 'decoder.block.9.layer.0.SelfAttention.o.weight', 'decoder.block.5.layer.0.SelfAttention.v.weight', 'decoder.block.17.layer.2.DenseReluDense.wo.weight', 'decoder.block.16.layer.0.SelfAttention.v.weight', 'decoder.block.11.layer.2.DenseReluDense.wi.weight', 'decoder.block.5.layer.0.SelfAttention.k.weight', 'decoder.block.4.layer.0.layer_norm.weight', 'decoder.block.5.layer.1.EncDecAttention.o.weight', 'decoder.block.16.layer.1.EncDecAttention.k.weight', 'decoder.block.5.layer.1.EncDecAttention.v.weight', 'decoder.block.1.layer.2.layer_norm.weight', 'decoder.block.13.layer.1.EncDecAttention.o.weight', 'decoder.block.15.layer.2.DenseReluDense.wi.weight', 'decoder.block.8.layer.0.SelfAttention.o.weight', 'decoder.block.0.layer.0.SelfAttention.o.weight', 'decoder.block.13.layer.0.layer_norm.weight', 'decoder.block.4.layer.0.SelfAttention.k.weight', 'decoder.block.18.layer.0.SelfAttention.k.weight', 'decoder.block.17.layer.1.EncDecAttention.o.weight', 'decoder.block.19.layer.1.EncDecAttention.k.weight', 'decoder.block.7.layer.2.DenseReluDense.wi.weight', 'decoder.block.23.layer.1.EncDecAttention.o.weight', 'decoder.block.12.layer.2.DenseReluDense.wo.weight', 'decoder.block.2.layer.0.SelfAttention.q.weight', 'decoder.block.15.layer.1.EncDecAttention.v.weight', 'decoder.block.22.layer.1.EncDecAttention.o.weight', 'decoder.block.19.layer.2.layer_norm.weight', 'decoder.block.8.layer.2.layer_norm.weight', 'decoder.block.23.layer.0.layer_norm.weight', 'decoder.block.15.layer.0.SelfAttention.v.weight', 'decoder.block.8.layer.1.EncDecAttention.q.weight', 'decoder.block.10.layer.1.EncDecAttention.v.weight', 'decoder.block.20.layer.1.EncDecAttention.q.weight', 'decoder.block.4.layer.2.DenseReluDense.wo.weight', 'decoder.block.0.layer.0.SelfAttention.q.weight', 'decoder.block.22.layer.2.layer_norm.weight', 'decoder.block.1.layer.0.SelfAttention.o.weight', 'lm_head.weight', 'decoder.block.8.layer.0.layer_norm.weight', 'decoder.block.3.layer.1.layer_norm.weight', 'decoder.block.8.layer.0.SelfAttention.v.weight', 'decoder.block.23.layer.1.EncDecAttention.v.weight', 'decoder.block.17.layer.1.EncDecAttention.k.weight', 'decoder.block.11.layer.1.EncDecAttention.k.weight', 'decoder.block.8.layer.1.EncDecAttention.k.weight', 'decoder.block.2.layer.2.DenseReluDense.wi.weight', 'decoder.block.1.layer.0.SelfAttention.k.weight', 'decoder.block.18.layer.2.DenseReluDense.wo.weight', 'decoder.block.2.layer.0.SelfAttention.v.weight', 'decoder.block.3.layer.1.EncDecAttention.k.weight', 'decoder.block.21.layer.0.SelfAttention.o.weight', 'decoder.block.4.layer.1.EncDecAttention.q.weight', 'decoder.block.11.layer.1.layer_norm.weight', 'decoder.block.23.layer.0.SelfAttention.v.weight', 'decoder.block.7.layer.0.layer_norm.weight', 'decoder.block.9.layer.2.DenseReluDense.wo.weight', 'decoder.block.16.layer.2.DenseReluDense.wi.weight', 'decoder.block.7.layer.0.SelfAttention.k.weight', 'decoder.block.10.layer.2.DenseReluDense.wi.weight', 'decoder.block.16.layer.1.EncDecAttention.q.weight', 'decoder.block.17.layer.0.SelfAttention.k.weight', 'decoder.block.1.layer.1.EncDecAttention.v.weight', 'decoder.block.6.layer.1.EncDecAttention.o.weight', 'decoder.block.18.layer.0.SelfAttention.q.weight', 'decoder.block.23.layer.2.DenseReluDense.wi.weight', 'decoder.block.5.layer.0.SelfAttention.q.weight', 'decoder.block.14.layer.0.SelfAttention.v.weight', 'decoder.block.16.layer.1.EncDecAttention.o.weight', 'decoder.block.23.layer.0.SelfAttention.q.weight', 'decoder.block.6.layer.2.DenseReluDense.wo.weight', 'decoder.block.13.layer.1.EncDecAttention.q.weight', 'decoder.block.16.layer.0.SelfAttention.q.weight', 'decoder.block.1.layer.0.layer_norm.weight', 'decoder.block.9.layer.0.SelfAttention.k.weight', 'decoder.block.18.layer.0.SelfAttention.v.weight', 'decoder.block.13.layer.2.DenseReluDense.wo.weight', 'decoder.block.15.layer.2.layer_norm.weight', 'decoder.block.13.layer.2.DenseReluDense.wi.weight', 'decoder.block.14.layer.1.EncDecAttention.v.weight', 'decoder.block.17.layer.1.EncDecAttention.v.weight', 'decoder.block.7.layer.0.SelfAttention.v.weight', 'decoder.block.17.layer.1.EncDecAttention.q.weight', 'decoder.block.22.layer.0.SelfAttention.q.weight', 'decoder.block.21.layer.1.EncDecAttention.o.weight', 'decoder.block.6.layer.2.DenseReluDense.wi.weight', 'decoder.block.9.layer.1.EncDecAttention.q.weight', 'decoder.block.8.layer.1.layer_norm.weight', 'decoder.block.19.layer.1.EncDecAttention.v.weight', 'decoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight', 'decoder.block.20.layer.0.SelfAttention.q.weight', 'decoder.block.7.layer.2.DenseReluDense.wo.weight', 'decoder.block.21.layer.0.SelfAttention.v.weight', 'decoder.block.8.layer.2.DenseReluDense.wi.weight', 'decoder.block.9.layer.0.SelfAttention.v.weight', 'decoder.final_layer_norm.weight', 'decoder.block.3.layer.1.EncDecAttention.q.weight', 'decoder.block.18.layer.1.EncDecAttention.v.weight', 'decoder.block.0.layer.1.EncDecAttention.o.weight', 'decoder.block.4.layer.2.layer_norm.weight', 'decoder.block.16.layer.2.DenseReluDense.wo.weight', 'decoder.block.17.layer.2.layer_norm.weight', 'decoder.block.18.layer.1.layer_norm.weight', 'decoder.block.1.layer.1.layer_norm.weight', 'decoder.block.9.layer.1.EncDecAttention.o.weight', 'decoder.block.11.layer.0.SelfAttention.q.weight', 'decoder.block.0.layer.0.layer_norm.weight', 'decoder.block.21.layer.0.SelfAttention.k.weight', 'decoder.block.14.layer.2.layer_norm.weight', 'decoder.block.4.layer.1.EncDecAttention.v.weight', 'decoder.block.3.layer.2.DenseReluDense.wo.weight', 'decoder.block.18.layer.2.layer_norm.weight', 'decoder.block.19.layer.1.EncDecAttention.o.weight', 'decoder.block.13.layer.0.SelfAttention.q.weight', 'decoder.block.7.layer.1.EncDecAttention.o.weight', 'decoder.block.13.layer.1.EncDecAttention.k.weight', 'decoder.block.2.layer.0.layer_norm.weight', 'decoder.block.13.layer.0.SelfAttention.k.weight', 'decoder.block.15.layer.2.DenseReluDense.wo.weight', 'decoder.block.22.layer.1.EncDecAttention.k.weight', 'decoder.block.12.layer.0.SelfAttention.q.weight', 'decoder.block.16.layer.1.EncDecAttention.v.weight', 'decoder.block.12.layer.1.EncDecAttention.o.weight', 'decoder.block.22.layer.2.DenseReluDense.wi.weight', 'decoder.block.21.layer.0.layer_norm.weight', 'decoder.block.19.layer.0.layer_norm.weight', 'decoder.block.11.layer.0.SelfAttention.o.weight', 'decoder.block.20.layer.2.layer_norm.weight', 'decoder.block.2.layer.1.EncDecAttention.q.weight', 'decoder.block.10.layer.2.DenseReluDense.wo.weight', 'decoder.block.18.layer.2.DenseReluDense.wi.weight', 'decoder.block.6.layer.0.SelfAttention.v.weight', 'decoder.block.20.layer.2.DenseReluDense.wi.weight', 'decoder.block.4.layer.1.layer_norm.weight', 'decoder.block.21.layer.1.layer_norm.weight', 'decoder.block.7.layer.1.layer_norm.weight', 'decoder.block.4.layer.0.SelfAttention.q.weight', 'decoder.block.6.layer.0.layer_norm.weight', 'decoder.block.22.layer.1.EncDecAttention.v.weight', 'decoder.block.3.layer.0.SelfAttention.k.weight', 'decoder.block.22.layer.1.EncDecAttention.q.weight', 'decoder.block.13.layer.0.SelfAttention.v.weight', 'decoder.block.10.layer.0.SelfAttention.v.weight', 'decoder.block.15.layer.1.EncDecAttention.k.weight', 'decoder.block.8.layer.0.SelfAttention.q.weight', 'decoder.block.10.layer.0.layer_norm.weight', 'decoder.block.14.layer.2.DenseReluDense.wo.weight', 'decoder.block.2.layer.2.DenseReluDense.wo.weight', 'decoder.block.21.layer.2.DenseReluDense.wo.weight', 'decoder.block.8.layer.2.DenseReluDense.wo.weight', 'decoder.block.20.layer.0.SelfAttention.k.weight', 'decoder.block.2.layer.1.layer_norm.weight', 'decoder.block.5.layer.1.EncDecAttention.q.weight', 'decoder.block.12.layer.2.DenseReluDense.wi.weight', 'decoder.block.22.layer.0.SelfAttention.o.weight', 'decoder.block.17.layer.0.SelfAttention.q.weight', 'decoder.block.15.layer.0.layer_norm.weight', 'decoder.block.4.layer.0.SelfAttention.v.weight', 'decoder.block.15.layer.1.layer_norm.weight', 'decoder.block.0.layer.1.layer_norm.weight', 'decoder.block.4.layer.0.SelfAttention.o.weight', 'decoder.block.6.layer.1.EncDecAttention.v.weight', 'decoder.block.3.layer.1.EncDecAttention.v.weight', 'decoder.block.14.layer.1.EncDecAttention.k.weight', 'decoder.block.22.layer.0.SelfAttention.v.weight', 'decoder.block.13.layer.0.SelfAttention.o.weight', 'decoder.block.7.layer.2.layer_norm.weight', 'decoder.block.17.layer.0.layer_norm.weight', 'decoder.block.3.layer.0.SelfAttention.v.weight', 'decoder.block.16.layer.2.layer_norm.weight', 'decoder.block.17.layer.0.SelfAttention.v.weight', 'decoder.block.10.layer.1.EncDecAttention.q.weight', 'decoder.block.17.layer.0.SelfAttention.o.weight', 'decoder.block.12.layer.1.EncDecAttention.v.weight', 'decoder.block.5.layer.2.DenseReluDense.wi.weight', 'decoder.block.20.layer.1.EncDecAttention.o.weight', 'decoder.block.21.layer.1.EncDecAttention.q.weight', 'decoder.block.17.layer.1.layer_norm.weight', 'decoder.block.19.layer.0.SelfAttention.q.weight', 'decoder.block.18.layer.1.EncDecAttention.k.weight', 'decoder.block.16.layer.0.SelfAttention.k.weight', 'decoder.block.3.layer.0.SelfAttention.q.weight', 'decoder.block.12.layer.2.layer_norm.weight', 'decoder.block.0.layer.2.DenseReluDense.wi.weight', 'decoder.block.0.layer.1.EncDecAttention.k.weight', 'decoder.block.21.layer.0.SelfAttention.q.weight', 'decoder.block.0.layer.0.SelfAttention.k.weight', 'decoder.block.18.layer.0.layer_norm.weight', 'decoder.block.9.layer.0.layer_norm.weight', 'decoder.block.9.layer.0.SelfAttention.q.weight', 'decoder.block.11.layer.0.layer_norm.weight', 'decoder.block.14.layer.1.layer_norm.weight', 'decoder.block.5.layer.0.SelfAttention.o.weight', 'decoder.block.10.layer.1.EncDecAttention.o.weight', 'decoder.block.13.layer.1.layer_norm.weight', 'decoder.block.12.layer.1.EncDecAttention.q.weight', 'decoder.block.23.layer.2.layer_norm.weight', 'decoder.block.5.layer.1.layer_norm.weight', 'decoder.block.18.layer.0.SelfAttention.o.weight', 'decoder.block.1.layer.1.EncDecAttention.q.weight', 'decoder.block.0.layer.2.DenseReluDense.wo.weight', 'decoder.block.7.layer.1.EncDecAttention.q.weight', 'decoder.block.20.layer.0.SelfAttention.v.weight', 'decoder.block.0.layer.0.SelfAttention.v.weight', 'decoder.block.3.layer.0.layer_norm.weight', 'decoder.block.14.layer.0.SelfAttention.q.weight', 'decoder.block.16.layer.0.SelfAttention.o.weight', 'decoder.block.5.layer.1.EncDecAttention.k.weight', 'decoder.block.10.layer.1.layer_norm.weight', 'decoder.block.4.layer.1.EncDecAttention.k.weight', 'decoder.block.6.layer.1.layer_norm.weight', 'decoder.block.19.layer.0.SelfAttention.v.weight', 'decoder.block.19.layer.0.SelfAttention.o.weight', 'decoder.block.1.layer.2.DenseReluDense.wo.weight', 'decoder.block.16.layer.0.layer_norm.weight', 'decoder.block.19.layer.1.layer_norm.weight', 'decoder.block.2.layer.2.layer_norm.weight', 'decoder.block.11.layer.2.layer_norm.weight', 'decoder.block.3.layer.1.EncDecAttention.o.weight', 'decoder.block.12.layer.0.layer_norm.weight', 'decoder.block.1.layer.0.SelfAttention.v.weight', 'decoder.block.2.layer.0.SelfAttention.k.weight', 'decoder.block.21.layer.2.layer_norm.weight', 'decoder.block.5.layer.0.layer_norm.weight', 'decoder.block.8.layer.1.EncDecAttention.o.weight', 'decoder.block.20.layer.0.layer_norm.weight', 'decoder.block.5.layer.2.DenseReluDense.wo.weight', 'decoder.block.23.layer.0.SelfAttention.o.weight', 'decoder.block.9.layer.2.layer_norm.weight', 'decoder.block.14.layer.1.EncDecAttention.q.weight', 'decoder.block.0.layer.2.layer_norm.weight', 'decoder.block.15.layer.1.EncDecAttention.q.weight', 'decoder.block.15.layer.0.SelfAttention.q.weight', 'decoder.block.3.layer.2.layer_norm.weight', 'decoder.block.6.layer.1.EncDecAttention.q.weight', 'decoder.block.23.layer.1.EncDecAttention.k.weight', 'decoder.block.3.layer.0.SelfAttention.o.weight', 'decoder.block.8.layer.0.SelfAttention.k.weight', 'decoder.block.4.layer.2.DenseReluDense.wi.weight', 'decoder.block.23.layer.1.layer_norm.weight', 'decoder.block.12.layer.1.EncDecAttention.k.weight', 'decoder.block.0.layer.1.EncDecAttention.q.weight', 'decoder.block.6.layer.0.SelfAttention.q.weight', 'decoder.block.4.layer.1.EncDecAttention.o.weight', 'decoder.block.18.layer.1.EncDecAttention.q.weight', 'decoder.block.14.layer.0.layer_norm.weight', 'decoder.block.11.layer.0.SelfAttention.k.weight', 'decoder.block.10.layer.0.SelfAttention.o.weight', 'decoder.block.19.layer.2.DenseReluDense.wi.weight', 'decoder.block.22.layer.2.DenseReluDense.wo.weight', 'decoder.block.20.layer.2.DenseReluDense.wo.weight', 'decoder.block.21.layer.2.DenseReluDense.wi.weight', 'decoder.block.7.layer.0.SelfAttention.q.weight', 'decoder.block.12.layer.0.SelfAttention.v.weight', 'decoder.block.18.layer.1.EncDecAttention.o.weight', 'decoder.block.1.layer.1.EncDecAttention.o.weight', 'decoder.block.22.layer.0.SelfAttention.k.weight', 'decoder.block.11.layer.1.EncDecAttention.o.weight', 'decoder.block.2.layer.0.SelfAttention.o.weight', 'decoder.block.1.layer.1.EncDecAttention.k.weight', 'decoder.block.22.layer.1.layer_norm.weight', 'decoder.block.14.layer.0.SelfAttention.o.weight', 'decoder.block.1.layer.2.DenseReluDense.wi.weight', 'decoder.block.7.layer.1.EncDecAttention.v.weight', 'decoder.block.11.layer.1.EncDecAttention.v.weight', 'decoder.block.8.layer.1.EncDecAttention.v.weight', 'decoder.block.13.layer.2.layer_norm.weight', 'decoder.block.15.layer.1.EncDecAttention.o.weight', 'decoder.block.20.layer.0.SelfAttention.o.weight', 'decoder.block.14.layer.1.EncDecAttention.o.weight', 'decoder.block.6.layer.2.layer_norm.weight', 'decoder.block.23.layer.1.EncDecAttention.q.weight', 'decoder.block.12.layer.0.SelfAttention.k.weight', 'decoder.block.19.layer.2.DenseReluDense.wo.weight', 'decoder.block.10.layer.1.EncDecAttention.k.weight', 'decoder.block.10.layer.2.layer_norm.weight', 'decoder.block.17.layer.2.DenseReluDense.wi.weight', 'decoder.block.15.layer.0.SelfAttention.k.weight', 'decoder.block.6.layer.1.EncDecAttention.k.weight', 'decoder.block.20.layer.1.layer_norm.weight', 'decoder.block.10.layer.0.SelfAttention.q.weight', 'decoder.block.11.layer.0.SelfAttention.v.weight', 'decoder.block.2.layer.1.EncDecAttention.o.weight', 'decoder.block.7.layer.0.SelfAttention.o.weight', 'decoder.block.9.layer.2.DenseReluDense.wi.weight', 'decoder.block.12.layer.1.layer_norm.weight', 'decoder.block.20.layer.1.EncDecAttention.v.weight']\n",
      "- This IS expected if you are initializing T5EncoderModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing T5EncoderModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ProtT5_Classfier\n",
      "Trainable Parameter: 1208142849\n",
      "ProtT5_LoRA_Classfier\n",
      "Trainable Parameter: 2508801\n",
      "\n",
      "[2023-10-20 16:11:39,001] [INFO] [comm.py:657:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl\n",
      "[2023-10-20 16:11:39,053] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed info: version=0.8.1, git-hash=unknown, git-branch=unknown\n",
      "[2023-10-20 16:11:41,123] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using /homes/schmirx6/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...\n",
      "Detected CUDA files, patching ldflags\n",
      "Emitting ninja build file /homes/schmirx6/.cache/torch_extensions/py39_cu117/cpu_adam/build.ninja...\n",
      "Building extension module cpu_adam...\n",
      "Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)\n",
      "Loading extension module cpu_adam...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ninja: no work to do.\n",
      "Time to load cpu_adam op: 3.048384428024292 seconds\n",
      "[2023-10-20 16:11:47,034] [INFO] [logging.py:75:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer\n",
      "[2023-10-20 16:11:47,071] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam\n",
      "[2023-10-20 16:11:47,071] [INFO] [utils.py:53:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>\n",
      "[2023-10-20 16:11:47,072] [INFO] [logging.py:75:log_dist] [Rank 0] Creating torch.float32 ZeRO stage 2 optimizer\n",
      "[2023-10-20 16:11:47,073] [INFO] [stage_1_and_2.py:144:__init__] Reduce bucket size 200000000\n",
      "[2023-10-20 16:11:47,073] [INFO] [stage_1_and_2.py:145:__init__] Allgather bucket size 200000000\n",
      "[2023-10-20 16:11:47,074] [INFO] [stage_1_and_2.py:146:__init__] CPU Offload: True\n",
      "[2023-10-20 16:11:47,074] [INFO] [stage_1_and_2.py:147:__init__] Round robin gradient partitioning: False\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using /homes/schmirx6/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...\n",
      "Emitting ninja build file /homes/schmirx6/.cache/torch_extensions/py39_cu117/utils/build.ninja...\n",
      "Building extension module utils...\n",
      "Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)\n",
      "Loading extension module utils...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ninja: no work to do.\n",
      "Time to load utils op: 0.5580213069915771 seconds\n",
      "Rank: 0 partition count [1] and sizes[(2508802, False)] \n",
      "[2023-10-20 16:11:47,892] [INFO] [utils.py:825:see_memory_usage] Before initializing optimizer states\n",
      "[2023-10-20 16:11:47,893] [INFO] [utils.py:826:see_memory_usage] MA 4.51 GB         Max_MA 4.51 GB         CA 4.52 GB         Max_CA 5 GB \n",
      "[2023-10-20 16:11:47,894] [INFO] [utils.py:834:see_memory_usage] CPU Virtual Memory:  used = 51.81 GB, percent = 27.8%\n",
      "[2023-10-20 16:11:48,079] [INFO] [utils.py:825:see_memory_usage] After initializing optimizer states\n",
      "[2023-10-20 16:11:48,080] [INFO] [utils.py:826:see_memory_usage] MA 4.51 GB         Max_MA 4.51 GB         CA 4.52 GB         Max_CA 5 GB \n",
      "[2023-10-20 16:11:48,081] [INFO] [utils.py:834:see_memory_usage] CPU Virtual Memory:  used = 51.84 GB, percent = 27.8%\n",
      "[2023-10-20 16:11:48,082] [INFO] [stage_1_and_2.py:527:__init__] optimizer state initialized\n",
      "[2023-10-20 16:11:48,235] [INFO] [utils.py:825:see_memory_usage] After initializing ZeRO optimizer\n",
      "[2023-10-20 16:11:48,236] [INFO] [utils.py:826:see_memory_usage] MA 4.51 GB         Max_MA 4.51 GB         CA 4.52 GB         Max_CA 5 GB \n",
      "[2023-10-20 16:11:48,237] [INFO] [utils.py:834:see_memory_usage] CPU Virtual Memory:  used = 51.84 GB, percent = 27.8%\n",
      "[2023-10-20 16:11:48,247] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw\n",
      "[2023-10-20 16:11:48,247] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed using configured LR scheduler = WarmupLR\n",
      "[2023-10-20 16:11:48,248] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed LR Scheduler = <deepspeed.runtime.lr_schedules.WarmupLR object at 0x7f1642324190>\n",
      "[2023-10-20 16:11:48,248] [INFO] [logging.py:75:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0003], mom=[[0.9, 0.999]]\n",
      "[2023-10-20 16:11:48,249] [INFO] [config.py:1009:print] DeepSpeedEngine configuration:\n",
      "[2023-10-20 16:11:48,250] [INFO] [config.py:1013:print]   activation_checkpointing_config  {\n",
      "    \"partition_activations\": false, \n",
      "    \"contiguous_memory_optimization\": false, \n",
      "    \"cpu_checkpointing\": false, \n",
      "    \"number_checkpoints\": null, \n",
      "    \"synchronize_checkpoint_boundary\": false, \n",
      "    \"profile\": false\n",
      "}\n",
      "[2023-10-20 16:11:48,251] [INFO] [config.py:1013:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}\n",
      "[2023-10-20 16:11:48,251] [INFO] [config.py:1013:print]   amp_enabled .................. False\n",
      "[2023-10-20 16:11:48,251] [INFO] [config.py:1013:print]   amp_params ................... False\n",
      "[2023-10-20 16:11:48,252] [INFO] [config.py:1013:print]   autotuning_config ............ {\n",
      "    \"enabled\": false, \n",
      "    \"start_step\": null, \n",
      "    \"end_step\": null, \n",
      "    \"metric_path\": null, \n",
      "    \"arg_mappings\": null, \n",
      "    \"metric\": \"throughput\", \n",
      "    \"model_info\": null, \n",
      "    \"results_dir\": \"autotuning_results\", \n",
      "    \"exps_dir\": \"autotuning_exps\", \n",
      "    \"overwrite\": true, \n",
      "    \"fast\": true, \n",
      "    \"start_profile_step\": 3, \n",
      "    \"end_profile_step\": 5, \n",
      "    \"tuner_type\": \"gridsearch\", \n",
      "    \"tuner_early_stopping\": 5, \n",
      "    \"tuner_num_trials\": 50, \n",
      "    \"model_info_path\": null, \n",
      "    \"mp_size\": 1, \n",
      "    \"max_train_batch_size\": null, \n",
      "    \"min_train_batch_size\": 1, \n",
      "    \"max_train_micro_batch_size_per_gpu\": 1.024000e+03, \n",
      "    \"min_train_micro_batch_size_per_gpu\": 1, \n",
      "    \"num_tuning_micro_batch_sizes\": 3\n",
      "}\n",
      "[2023-10-20 16:11:48,252] [INFO] [config.py:1013:print]   bfloat16_enabled ............. False\n",
      "[2023-10-20 16:11:48,253] [INFO] [config.py:1013:print]   checkpoint_parallel_write_pipeline  False\n",
      "[2023-10-20 16:11:48,253] [INFO] [config.py:1013:print]   checkpoint_tag_validation_enabled  True\n",
      "[2023-10-20 16:11:48,253] [INFO] [config.py:1013:print]   checkpoint_tag_validation_fail  False\n",
      "[2023-10-20 16:11:48,254] [INFO] [config.py:1013:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7f1642324850>\n",
      "[2023-10-20 16:11:48,254] [INFO] [config.py:1013:print]   communication_data_type ...... None\n",
      "[2023-10-20 16:11:48,254] [INFO] [config.py:1013:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}\n",
      "[2023-10-20 16:11:48,255] [INFO] [config.py:1013:print]   curriculum_enabled_legacy .... False\n",
      "[2023-10-20 16:11:48,255] [INFO] [config.py:1013:print]   curriculum_params_legacy ..... False\n",
      "[2023-10-20 16:11:48,256] [INFO] [config.py:1013:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}\n",
      "[2023-10-20 16:11:48,256] [INFO] [config.py:1013:print]   data_efficiency_enabled ...... False\n",
      "[2023-10-20 16:11:48,256] [INFO] [config.py:1013:print]   dataloader_drop_last ......... False\n",
      "[2023-10-20 16:11:48,257] [INFO] [config.py:1013:print]   disable_allgather ............ False\n",
      "[2023-10-20 16:11:48,257] [INFO] [config.py:1013:print]   dump_state ................... False\n",
      "[2023-10-20 16:11:48,257] [INFO] [config.py:1013:print]   dynamic_loss_scale_args ...... None\n",
      "[2023-10-20 16:11:48,258] [INFO] [config.py:1013:print]   eigenvalue_enabled ........... False\n",
      "[2023-10-20 16:11:48,258] [INFO] [config.py:1013:print]   eigenvalue_gas_boundary_resolution  1\n",
      "[2023-10-20 16:11:48,258] [INFO] [config.py:1013:print]   eigenvalue_layer_name ........ bert.encoder.layer\n",
      "[2023-10-20 16:11:48,259] [INFO] [config.py:1013:print]   eigenvalue_layer_num ......... 0\n",
      "[2023-10-20 16:11:48,259] [INFO] [config.py:1013:print]   eigenvalue_max_iter .......... 100\n",
      "[2023-10-20 16:11:48,259] [INFO] [config.py:1013:print]   eigenvalue_stability ......... 1e-06\n",
      "[2023-10-20 16:11:48,260] [INFO] [config.py:1013:print]   eigenvalue_tol ............... 0.01\n",
      "[2023-10-20 16:11:48,260] [INFO] [config.py:1013:print]   eigenvalue_verbose ........... False\n",
      "[2023-10-20 16:11:48,260] [INFO] [config.py:1013:print]   elasticity_enabled ........... False\n",
      "[2023-10-20 16:11:48,261] [INFO] [config.py:1013:print]   flops_profiler_config ........ {\n",
      "    \"enabled\": false, \n",
      "    \"profile_step\": 1, \n",
      "    \"module_depth\": -1, \n",
      "    \"top_modules\": 1, \n",
      "    \"detailed\": true, \n",
      "    \"output_file\": null\n",
      "}\n",
      "[2023-10-20 16:11:48,261] [INFO] [config.py:1013:print]   fp16_auto_cast ............... None\n",
      "[2023-10-20 16:11:48,262] [INFO] [config.py:1013:print]   fp16_enabled ................. False\n",
      "[2023-10-20 16:11:48,262] [INFO] [config.py:1013:print]   fp16_master_weights_and_gradients  False\n",
      "[2023-10-20 16:11:48,262] [INFO] [config.py:1013:print]   global_rank .................. 0\n",
      "[2023-10-20 16:11:48,263] [INFO] [config.py:1013:print]   grad_accum_dtype ............. None\n",
      "[2023-10-20 16:11:48,263] [INFO] [config.py:1013:print]   gradient_accumulation_steps .. 1\n",
      "[2023-10-20 16:11:48,263] [INFO] [config.py:1013:print]   gradient_clipping ............ 1.0\n",
      "[2023-10-20 16:11:48,264] [INFO] [config.py:1013:print]   gradient_predivide_factor .... 1.0\n",
      "[2023-10-20 16:11:48,264] [INFO] [config.py:1013:print]   initial_dynamic_scale ........ 65536\n",
      "[2023-10-20 16:11:48,264] [INFO] [config.py:1013:print]   load_universal_checkpoint .... False\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-10-20 16:11:48,265] [INFO] [config.py:1013:print]   loss_scale ................... 0\n",
      "[2023-10-20 16:11:48,265] [INFO] [config.py:1013:print]   memory_breakdown ............. False\n",
      "[2023-10-20 16:11:48,266] [INFO] [config.py:1013:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False\n",
      "[2023-10-20 16:11:48,266] [INFO] [config.py:1013:print]   nebula_config ................ {\n",
      "    \"enabled\": false, \n",
      "    \"persistent_storage_path\": null, \n",
      "    \"persistent_time_interval\": 100, \n",
      "    \"num_of_version_in_retention\": 2, \n",
      "    \"enable_nebula_load\": true, \n",
      "    \"load_path\": null\n",
      "}\n",
      "[2023-10-20 16:11:48,266] [INFO] [config.py:1013:print]   optimizer_legacy_fusion ...... False\n",
      "[2023-10-20 16:11:48,267] [INFO] [config.py:1013:print]   optimizer_name ............... adamw\n",
      "[2023-10-20 16:11:48,267] [INFO] [config.py:1013:print]   optimizer_params ............. {'lr': 0.0003, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.0}\n",
      "[2023-10-20 16:11:48,267] [INFO] [config.py:1013:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}\n",
      "[2023-10-20 16:11:48,268] [INFO] [config.py:1013:print]   pld_enabled .................. False\n",
      "[2023-10-20 16:11:48,268] [INFO] [config.py:1013:print]   pld_params ................... False\n",
      "[2023-10-20 16:11:48,269] [INFO] [config.py:1013:print]   prescale_gradients ........... False\n",
      "[2023-10-20 16:11:48,269] [INFO] [config.py:1013:print]   scheduler_name ............... WarmupLR\n",
      "[2023-10-20 16:11:48,269] [INFO] [config.py:1013:print]   scheduler_params ............. {'warmup_min_lr': 0, 'warmup_max_lr': 0.0003, 'warmup_num_steps': 0}\n",
      "[2023-10-20 16:11:48,270] [INFO] [config.py:1013:print]   sparse_attention ............. None\n",
      "[2023-10-20 16:11:48,270] [INFO] [config.py:1013:print]   sparse_gradients_enabled ..... False\n",
      "[2023-10-20 16:11:48,270] [INFO] [config.py:1013:print]   steps_per_print .............. 2000\n",
      "[2023-10-20 16:11:48,271] [INFO] [config.py:1013:print]   train_batch_size ............. 1\n",
      "[2023-10-20 16:11:48,271] [INFO] [config.py:1013:print]   train_micro_batch_size_per_gpu  1\n",
      "[2023-10-20 16:11:48,271] [INFO] [config.py:1013:print]   use_node_local_storage ....... False\n",
      "[2023-10-20 16:11:48,272] [INFO] [config.py:1013:print]   wall_clock_breakdown ......... False\n",
      "[2023-10-20 16:11:48,272] [INFO] [config.py:1013:print]   world_size ................... 1\n",
      "[2023-10-20 16:11:48,272] [INFO] [config.py:1013:print]   zero_allow_untested_optimizer  False\n",
      "[2023-10-20 16:11:48,273] [INFO] [config.py:1013:print]   zero_config .................. stage=2 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=200000000 allgather_partitions=True allgather_bucket_size=200000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='cpu', nvme_path=None, buffer_count=4, pin_memory=True, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False) sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False\n",
      "[2023-10-20 16:11:48,273] [INFO] [config.py:1013:print]   zero_enabled ................. True\n",
      "[2023-10-20 16:11:48,274] [INFO] [config.py:1013:print]   zero_optimization_stage ...... 2\n",
      "[2023-10-20 16:11:48,274] [INFO] [config.py:998:print_user_config]   json = {\n",
      "    \"fp16\": {\n",
      "        \"enabled\": false, \n",
      "        \"loss_scale\": 0, \n",
      "        \"loss_scale_window\": 1000, \n",
      "        \"initial_scale_power\": 16, \n",
      "        \"hysteresis\": 2, \n",
      "        \"min_loss_scale\": 1\n",
      "    }, \n",
      "    \"optimizer\": {\n",
      "        \"type\": \"AdamW\", \n",
      "        \"params\": {\n",
      "            \"lr\": 0.0003, \n",
      "            \"betas\": [0.9, 0.999], \n",
      "            \"eps\": 1e-08, \n",
      "            \"weight_decay\": 0.0\n",
      "        }\n",
      "    }, \n",
      "    \"scheduler\": {\n",
      "        \"type\": \"WarmupLR\", \n",
      "        \"params\": {\n",
      "            \"warmup_min_lr\": 0, \n",
      "            \"warmup_max_lr\": 0.0003, \n",
      "            \"warmup_num_steps\": 0\n",
      "        }\n",
      "    }, \n",
      "    \"zero_optimization\": {\n",
      "        \"stage\": 2, \n",
      "        \"offload_optimizer\": {\n",
      "            \"device\": \"cpu\", \n",
      "            \"pin_memory\": true\n",
      "        }, \n",
      "        \"allgather_partitions\": true, \n",
      "        \"allgather_bucket_size\": 2.000000e+08, \n",
      "        \"overlap_comm\": true, \n",
      "        \"reduce_scatter\": true, \n",
      "        \"reduce_bucket_size\": 2.000000e+08, \n",
      "        \"contiguous_gradients\": true\n",
      "    }, \n",
      "    \"gradient_accumulation_steps\": 1, \n",
      "    \"gradient_clipping\": 1.0, \n",
      "    \"steps_per_print\": 2.000000e+03, \n",
      "    \"train_batch_size\": 1, \n",
      "    \"train_micro_batch_size_per_gpu\": 1, \n",
      "    \"wall_clock_breakdown\": false\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using /homes/schmirx6/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...\n",
      "No modifications detected for re-loaded extension module utils, skipping build step...\n",
      "Loading extension module utils...\n",
      "***** Running training *****\n",
      "  Num examples = 1056\n",
      "  Num Epochs = 10\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time to load utils op: 0.006617546081542969 seconds\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  Instantaneous batch size per device = 1\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 1\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 10560\n",
      "  Number of trainable parameters = 2508801\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adam Optimizer #0 is created with AVX2 arithmetic capability.\n",
      "Config: alpha=0.000300, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_w=1\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='8977' max='10560' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [ 8977/10560 1:08:57 < 12:09, 2.17 it/s, Epoch 8.50/10]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Spearmanr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>528</td>\n",
       "      <td>No log</td>\n",
       "      <td>10.112532</td>\n",
       "      <td>0.743306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1056</td>\n",
       "      <td>No log</td>\n",
       "      <td>9.466753</td>\n",
       "      <td>0.778502</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1584</td>\n",
       "      <td>25.888700</td>\n",
       "      <td>8.979529</td>\n",
       "      <td>0.788572</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2112</td>\n",
       "      <td>25.888700</td>\n",
       "      <td>9.223991</td>\n",
       "      <td>0.791704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2640</td>\n",
       "      <td>8.247300</td>\n",
       "      <td>8.695718</td>\n",
       "      <td>0.793680</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3168</td>\n",
       "      <td>8.247300</td>\n",
       "      <td>8.642170</td>\n",
       "      <td>0.796580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3696</td>\n",
       "      <td>6.848400</td>\n",
       "      <td>9.410137</td>\n",
       "      <td>0.800314</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4224</td>\n",
       "      <td>6.848400</td>\n",
       "      <td>8.512393</td>\n",
       "      <td>0.803212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4752</td>\n",
       "      <td>6.022700</td>\n",
       "      <td>9.152840</td>\n",
       "      <td>0.800926</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5280</td>\n",
       "      <td>6.022700</td>\n",
       "      <td>8.346140</td>\n",
       "      <td>0.802669</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5808</td>\n",
       "      <td>5.489600</td>\n",
       "      <td>8.859254</td>\n",
       "      <td>0.806098</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6336</td>\n",
       "      <td>5.489600</td>\n",
       "      <td>8.396515</td>\n",
       "      <td>0.806989</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6864</td>\n",
       "      <td>4.939900</td>\n",
       "      <td>9.113558</td>\n",
       "      <td>0.803928</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7392</td>\n",
       "      <td>4.939900</td>\n",
       "      <td>8.746425</td>\n",
       "      <td>0.809373</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7920</td>\n",
       "      <td>4.532800</td>\n",
       "      <td>8.656758</td>\n",
       "      <td>0.805631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8448</td>\n",
       "      <td>4.532800</td>\n",
       "      <td>8.011671</td>\n",
       "      <td>0.801585</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>\n",
       "    <div>\n",
       "      \n",
       "      <progress value='73' max='118' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [ 73/118 00:07 < 00:04, 9.79 it/s]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 118\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 118\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 118\n",
      "  Batch size = 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-10-20 16:27:03,319] [INFO] [logging.py:75:log_dist] [Rank 0] step=2000, skipped=0, lr=[0.0003], mom=[[0.9, 0.999]]\n",
      "[2023-10-20 16:27:03,320] [INFO] [timer.py:198:stop] epoch=0/micro_step=2000/global_step=2000, RunningAvgSamplesPerSec=2.3109632529959456, CurrSamplesPerSec=2.3104339323831162, MemAllocated=4.51GB, MaxMemAllocated=12.49GB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 118\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 118\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 118\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 118\n",
      "  Batch size = 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-10-20 16:42:32,014] [INFO] [logging.py:75:log_dist] [Rank 0] step=4000, skipped=0, lr=[0.0003], mom=[[0.9, 0.999]]\n",
      "[2023-10-20 16:42:32,015] [INFO] [timer.py:198:stop] epoch=0/micro_step=4000/global_step=4000, RunningAvgSamplesPerSec=2.3087094168932545, CurrSamplesPerSec=2.3053124972518533, MemAllocated=4.51GB, MaxMemAllocated=12.49GB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 118\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 118\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 118\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 118\n",
      "  Batch size = 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-10-20 16:57:59,247] [INFO] [logging.py:75:log_dist] [Rank 0] step=6000, skipped=0, lr=[0.0003], mom=[[0.9, 0.999]]\n",
      "[2023-10-20 16:57:59,248] [INFO] [timer.py:198:stop] epoch=0/micro_step=6000/global_step=6000, RunningAvgSamplesPerSec=2.309190655991188, CurrSamplesPerSec=2.309863901933173, MemAllocated=4.51GB, MaxMemAllocated=12.49GB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 118\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 118\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 118\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 118\n",
      "  Batch size = 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-10-20 17:13:25,665] [INFO] [logging.py:75:log_dist] [Rank 0] step=8000, skipped=0, lr=[0.0003], mom=[[0.9, 0.999]]\n",
      "[2023-10-20 17:13:25,666] [INFO] [timer.py:198:stop] epoch=0/micro_step=8000/global_step=8000, RunningAvgSamplesPerSec=2.309880076872569, CurrSamplesPerSec=2.3122349377963474, MemAllocated=4.51GB, MaxMemAllocated=12.49GB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 118\n",
      "  Batch size = 1\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 118\n",
      "  Batch size = 1\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tokenizer, model, history = train_per_residue(train, valid, num_labels=1, batch=1, accum=1, epochs=10, seed=42, gpu=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54bab485",
   "metadata": {},
   "source": [
    "## Plot results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "f8465267",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA30AAAHWCAYAAADDzuC9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAACG7klEQVR4nO3deVhU9f4H8PeZGWbY930REFDBPRdUyFwos6JMrTRLba+rlVm/0nLLFtPSvC3qbdMWzcqb1i2zlNJyzwXNBRRBAdlB9n1mfn8cZmBkERA4s7xfz3MeZs6cOfMZRDjv+W6CVqvVgoiIiIiIiMySTOoCiIiIiIiIqPMw9BEREREREZkxhj4iIiIiIiIzxtBHRERERERkxhj6iIiIiIiIzBhDHxERERERkRlj6CMiIiIiIjJjDH1ERERERERmjKGPiIiIiIjIjDH0ERFRh5k5cyaCgoLa9dwlS5ZAEISOLaiTXLx4EYIgYMOGDVKXQkREdE0MfUREFkAQhFZtu3fvlrpUScycORP29vbNPi4IAmbPnn3dr7NmzRoGRSIi6nIKqQsgIqLO9+WXXxrc/+KLL7Bz585G+8PDw6/rdT7++GNoNJp2PXfBggWYN2/edb1+VwkMDERFRQWsrKza9Lw1a9bA3d0dM2fO7JzCiIiImsDQR0RkAR544AGD+wcPHsTOnTsb7b9aeXk5bG1tW/06bQ1BDSkUCigUpvFnSRAEWFtbS10GAKCyshJKpRIyGTvvEBFR0/gXgoiIAACjRo1Cnz59cPToUYwcORK2trZ4+eWXAQA//PADbr/9dvj6+kKlUiEkJASvvfYa1Gq1wTmuHtOnG/v2zjvv4KOPPkJISAhUKhWGDBmCv//+2+C5TY3p03Wr3LZtG/r06QOVSoXevXtjx44djerfvXs3Bg8eDGtra4SEhOA///lPp40TbGpMX1ZWFh566CH4+/tDpVLBx8cHd911Fy5evAgACAoKwunTp7Fnzx59d9pRo0bpn5+cnIx77rkHrq6usLW1xbBhw/Dzzz83eo+CIGDz5s1YsGAB/Pz8YGtri/j4eAiCgHfffbdRrfv374cgCPj66687/PtARESmwTQ+UiUioi6Rn5+P8ePHY8qUKXjggQfg5eUFANiwYQPs7e0xd+5c2Nvb4/fff8eiRYtQXFyMt99++5rn3bRpE0pKSvDEE09AEASsWLECEydORHJy8jVbB/fu3Yvvv/8e//rXv+Dg4ID33nsPkyZNQmpqKtzc3AAAx48fx6233gofHx+8+uqrUKvVWLp0KTw8PNr0/vPy8tp0fEOTJk3C6dOn8fTTTyMoKAg5OTnYuXMnUlNTERQUhNWrV+Ppp5+Gvb09XnnlFQDQf3+zs7MxYsQIlJeX45lnnoGbmxs+//xz3HnnndiyZQvuvvtug9d67bXXoFQq8cILL6Cqqgq9evVCVFQUNm7ciOeee87g2I0bN8LBwQF33XVXu98bERGZOC0REVmcWbNmaa/+E3DTTTdpAWjXrVvX6Pjy8vJG+5544gmtra2ttrKyUr9vxowZ2sDAQP39lJQULQCtm5ubtqCgQL//hx9+0ALQ/u9//9PvW7x4caOaAGiVSqU2KSlJv+/EiRNaANr3339fvy82NlZra2urvXz5sn7f+fPntQqFotE5mzJjxgwtgBa3WbNmNXpf69ev12q1Wu2VK1e0ALRvv/12i6/Tu3dv7U033dRo/5w5c7QAtH/99Zd+X0lJiTY4OFgbFBSkVavVWq1Wq/3jjz+0ALTdu3dv9G/yn//8RwtAe/bsWf2+6upqrbu7u3bGjBnX/B4QEZH5YvdOIiLSU6lUeOihhxrtt7Gx0d8uKSlBXl4ebrzxRpSXlyMhIeGa573vvvvg4uKiv3/jjTcCELs0XktMTAxCQkL09/v16wdHR0f9c9VqNXbt2oUJEybA19dXf1xoaCjGjx9/zfPrWFtbY+fOnU1u12JjYwOlUondu3fjypUrrX5Nne3bt2Po0KGIjo7W77O3t8fjjz+Oixcv4syZMwbHz5gxw+DfBADuvfdeWFtbY+PGjfp9v/76K/Ly8q45dpOIiMwbu3cSEZGen58flEplo/2nT5/GggUL8Pvvv6O4uNjgsaKiomuet1u3bgb3dQGwNQHp6ufqnq97bk5ODioqKhAaGtrouKb2NUculyMmJqbVxzekUqmwfPlyPP/88/Dy8sKwYcNwxx13YPr06fD29r7m8y9duoTIyMhG+3WzqV66dAl9+vTR7w8ODm50rLOzM2JjY7Fp0ya89tprAMSunX5+fhgzZky73hcREZkHtvQREZHe1a1HAFBYWIibbroJJ06cwNKlS/G///0PO3fuxPLlywGgVUs0yOXyJvdrtdpOfW5XmjNnDs6dO4dly5bB2toaCxcuRHh4OI4fP97hr9XUvxMATJ8+HcnJydi/fz9KSkrw448/YurUqZzZk4jIwrGlj4iIWrR7927k5+fj+++/x8iRI/X7U1JSJKyqnqenJ6ytrZGUlNTosab2daaQkBA8//zzeP7553H+/HkMGDAAK1euxFdffQUAzc4kGhgYiMTExEb7dV1nAwMDW/X6t956Kzw8PLBx40ZERkaivLwcDz74YDvfDRERmQt+9EdERC3StbQ1bFmrrq7GmjVrpCrJgK5b5rZt25CRkaHfn5SUhF9++aVLaigvL0dlZaXBvpCQEDg4OKCqqkq/z87ODoWFhY2ef9ttt+Hw4cM4cOCAfl9ZWRk++ugjBAUFISIiolV1KBQKTJ06Fd9++y02bNiAvn37ol+/fu17U0REZDbY0kdERC0aMWIEXFxcMGPGDDzzzDMQBAFffvmlUXWvXLJkCX777TdERUXhqaeeglqtxgcffIA+ffogPj6+01//3LlzGDt2LO69915ERERAoVBg69atyM7OxpQpU/THDRo0CGvXrsXrr7+O0NBQeHp6YsyYMZg3bx6+/vprjB8/Hs888wxcXV3x+eefIyUlBf/973/b1D1z+vTpeO+99/DHH3/ou+ASEZFlY+gjIqIWubm54aeffsLzzz+PBQsWwMXFBQ888ADGjh2LcePGSV0eADFM/fLLL3jhhRewcOFCBAQEYOnSpTh79myrZhe9XgEBAZg6dSri4uLw5ZdfQqFQoFevXvj2228xadIk/XGLFi3CpUuXsGLFCpSUlOCmm27CmDFj4OXlhf379+Oll17C+++/j8rKSvTr1w//+9//cPvtt7eplkGDBqF37944e/Yspk2b1tFvlYiITJCgNaaPaomIiDrQhAkTcPr0aZw/f17qUrrUwIED4erqiri4OKlLISIiI8AxfUREZBYqKioM7p8/fx7bt2/HqFGjpClIIkeOHEF8fDymT58udSlERGQk2NJHRERmwcfHBzNnzkT37t1x6dIlrF27FlVVVTh+/DjCwsKkLq/TnTp1CkePHsXKlSuRl5eH5ORkWFtbS10WEREZAY7pIyIis3Drrbfi66+/RlZWFlQqFYYPH44333zTIgIfAGzZsgVLly5Fz5498fXXXzPwERGRHlv6iIiIiIiImvDnn3/i7bffxtGjR5GZmYmtW7diwoQJLT5n9+7dmDt3Lk6fPo2AgAAsWLAAM2fO7JJ6m8MxfURERERERE0oKytD//798eGHH7bq+JSUFNx+++0YPXo04uPjMWfOHDz66KP49ddfO7nSlrGlj4iIiIiI6BoEQbhmS99LL72En3/+GadOndLvmzJlCgoLC7Fjx44uqLJpZj+mr7a2FsePH4eXl1ebFrclIiIiIiLzotFokJqaioiICCgU9VFIpVJBpVJd9/kPHDiAmJgYg33jxo3DnDlzrvvc18PsQ9/x48cxdOhQqcsgIiIiIiIjtXjxYixZsuS6z5OVlQUvLy+DfV5eXiguLkZFRQVsbGyu+zXaw+xDn+6bfvjwYfj4+EhcDRERERERSSUzMxNDhw7FqVOnEBAQoN/fEa18xszsQ5+uS6ePjw/8/f0lroaIiIiIiKTm5OQER0fHDj+vt7c3srOzDfZlZ2fD0dFRslY+gLN3EhERERERdYjhw4cjLi7OYN/OnTsxfPhwiSoSMfQRERERERE1obS0FPHx8YiPjwcgLskQHx+P1NRUAMD8+fMxffp0/fFPPvkkkpOT8eKLLyIhIQFr1qzBt99+i+eee06K8vUY+oiIiIiIiJpw5MgRDBw4EAMHDgQAzJ07FwMHDsSiRYsAiGMEdQEQAIKDg/Hzzz9j586d6N+/P1auXIlPPvkE48aNk6R+HbNfpy89PR0BAQFIS0vjmD4iIiIiIgtmqdmALX1ERERERERmjKGPiIiIiIjIjDH0ERERERERmTGGPiIiIiIiIjPG0EdERERERGTGGPqIiIiIiIjMGEMfERERERGRGWPoIyIiIiIiMmMMfQSNRit1CURERERE1EkY+iyYWqPFrE3HMOj1ncgprpS6HCIiIiIi6gQMfRZMLhOQml+OK+U12HchT+pyiIiIiIioEzD0WbioUHcAwN7z+RJXQkREREREnYGhz8JF14W+fUl50Go5to+IiIiIyNww9Fm4wUEuUCpkyCquxIXcMqnLISIiIiKiDsbQZ+GsreQYEuQCQGztIyIiIiIi88LQR/Xj+hj6iIiIiIjMDkMf6cf1HbyQj1q1RuJqiIiIiIioIzH0EXr7OsHJxgolVbX453KR1OUQEREREVEHYugjyGUCRoS4AeC4PiIiIiIic8PQRwA4ro+IiIiIyFxJGvqWLVuGIUOGwMHBAZ6enpgwYQISExMNjhk1ahQEQTDYnnzySYkqNl+6cX3HLhWivLpW4mqIiIiIiKijSBr69uzZg1mzZuHgwYPYuXMnampqcMstt6CszHC9uMceewyZmZn6bcWKFRJVbL4C3Wzh52yDarUGf1+8InU5RERERETUQRRSvviOHTsM7m/YsAGenp44evQoRo4cqd9va2sLb2/vri7PogiCgOhQd3xzJA37kvJwUw8PqUsiIiIiIqIOYFRj+oqKxJkjXV1dDfZv3LgR7u7u6NOnD+bPn4/y8vJmz1FVVYXi4mL9VlJS0qk1m5OosLpxfec5ro+IiIiIyFxI2tLXkEajwZw5cxAVFYU+ffro999///0IDAyEr68vTp48iZdeegmJiYn4/vvvmzzPsmXL8Oqrr3ZV2WZFN4Pnmcxi5JdWwc1eJXFFRERERER0vQStVquVuggAeOqpp/DLL79g79698Pf3b/a433//HWPHjkVSUhJCQkIaPV5VVYWqqir9/cuXLyMiIgJpaWktnpdE4//9F85mFuP9qQMR299X6nKIiIiIiDpMeno6AgICLC4bGEX3ztmzZ+Onn37CH3/8cc1vfmRkJAAgKSmpycdVKhUcHR31m4ODQ4fXa86iuF4fEREREZFZkTT0abVazJ49G1u3bsXvv/+O4ODgaz4nPj4eAODj49PJ1Vkm3bi+v87nwUgagYmIiIiI6DpIOqZv1qxZ2LRpE3744Qc4ODggKysLAODk5AQbGxtcuHABmzZtwm233QY3NzecPHkSzz33HEaOHIl+/fpJWbrZGhrkCiu5gMuFFUgtKEegm53UJRERERER0XWQtKVv7dq1KCoqwqhRo+Dj46PfvvnmGwCAUqnErl27cMstt6BXr154/vnnMWnSJPzvf/+TsmyzZqdSYGA3FwDAXnbxJCIiIiIyeZK29F2r+2BAQAD27NnTRdWQTnSoOw6nFGBfUh6mRQZKXQ4REREREV0Ho5jIhYxLVKg4rm//hXyoNRzXR0RERERkyhj6qJH+/k6wVylQWF6DMxnFUpdDRERERETXgaGPGlHIZRjWvW7phgsc10dEREREZMoY+qhJ0aFcr4+IiIiIyBww9FGTouvW6zucUoDKGrXE1RARERERUXsx9FGTQjzs4eWoQlWtBscuXZG6HCIiIiIiaieGPmqSIAj6WTy5Xh8RERERkeli6KNmRdeFPo7rIyIiIiIyXQx91CxdS9/Jy0UoKq+RuBoiIiIiImoPhj5qlpejNcI87aHVAgeS2dpHRERERGSKGPqoRRzXR0RERERk2hj6qEVR+nF9+RJXQkRERERE7cHQRy2K7O4KuUxASl4Z0q+US10OERERERG1EUMftcjR2gr9/Z0AAPvZ2kdEREREZHIY+uiaojmuj4iIiIjIZDH00TVFNVivT6PRSlwNERERERG1BUMfXdPAbi6wsZIjv6waidklUpdDRERERERtwNBH16RUyBDZ3RWA2NpHRERERESmg6GPWiW6QRdPIiIiIiIyHQx91Cq6cX2HUgpQXauRuBoiIiIiImothj5qlZ5eDnC3V6K8Wo34tEKpyyEiIiIiolZi6KNWkckEjAjh0g1ERERERKaGoY9ajeP6iIiIiIhMD0MftVpUmBj64tMKUVJZI3E1RERERETUGgx91Gp+zjYIdreDWqPFoeQCqcshIiIiIqJWYOijNokKdQPAcX1ERERERKaCoY/aJCqE4/qIiIiIiEwJQx+1yfAQNwgCcD6nFNnFlVKXQ0RERERE18DQR23ibKtEXz8nAGztIyIiIiIyBQx91GZRoVyvj4iIiIjIVDD0UZs1XK9Pq9VKXA0REREREbWEoY/abFCgC1QKGbKLq3Aht1TqcoiIiIiIqAUMfdRm1lZyDAlyBQDsPc8unkRERERExoyhj9pFN65v34V8iSshIiIiIqKWKKQugExTdKg7lgM4eCEftWoNFHJ+fkBEJAmtVtwEQdzMmVYLlJUBxcVAUZH4teHt1nytrgZsbVvebGyufUxzz1Eqzf/fgYhMDkMftUuEryOcba1QWF6Dk5eLcEM3F6lLIiJqWW0tUFFRv5WXG95vamvumJoaQKMB1GrDr83dvtbj13OuhhNqqVRi+LC2bnprz2OtfY5c3vL3v6qq/UGt4VeNpnN/Tq6XTNb2oGhrC9jZiVvD283ts7KS+l2at+pqoLRU/IChrKzl2xUVhv8HdYG/YfBv6nZnPN7wq+7/59U/Z1fv03291v9fC/fhhx/i7bffRlZWFvr374/3338fQ4cObfb41atXY+3atUhNTYW7uzsmT56MZcuWwdraugurNsTQR+0ilwkYEeKG7f9kYd/5PIY+shxqNVBZKV7AVlZe3225XGwVUCrFi3Xd7bZuTT1XLpeutUGrFQNWba0Yjmpqmr59rcd1tysrWx/GWjqmtlaa70dXqqoSNykoFI0DoUZj2MLWUWQywMkJcHRs+1elUvyZKi+/vk33M1ZeLgYAtVqsTaMRw0BpJ050plC0HApbExxbum8KAUCtFr/3DUPYtQJaa29bwu+KqymVzQfClsJiWx6zsxO/mphvvvkGc+fOxbp16xAZGYnVq1dj3LhxSExMhKenZ6PjN23ahHnz5uGzzz7DiBEjcO7cOcycOROCIGDVqlUSvAMRQx+1W1SoO7b/k4W9SXl4emyY1OWQpdG12rT2Aq21gexax5nKxYAgtD0wWlmJga0tgaypfabwPdKFEt2muyhpaWt4jFIpBg+ZTLxAvt7bHXEO3QcSTW0VFc0/dq3HW3qspqb+e1pb27qwY2/f/sCm+2pra3xdKGtqri80lpUZ3m7qvi5Y1taKQbqoqHPei67FuLWtSq095nof02jqg15lZdvfV1splfVB2N6+6du2tuL/P6C+xe/qr129T6Op/7/b8Gft6q8Nv4fV1eLWWT9TADB2LLBrV+edv5OsWrUKjz32GB566CEAwLp16/Dzzz/js88+w7x58xodv3//fkRFReH+++8HAAQFBWHq1Kk4dOhQl9Z9NYY+ajfden3HUq+gvLoWtkr+OBHEPzZtCWNNXQC1Zmt4sSkVuby+VUOlMmzlaHi/qdsqlfi9qq4Ww6TuD25bt4bPbUirlbbVpylWVuKmULR8++p9Vwe0lsJYa45Rqeov0uj61NY2/tCkYVgExKCmC2sODqbRitQeVlb177Uz6D6QaSkUXu99XWgwtt8dLRGE5gPZ9dy2hG60unDYUjBs6bG2HKMbS2tESkpKUFxcrL+vUqmgUqkMjqmursbRo0cxf/58/T6ZTIaYmBgcOHCgyfOOGDECX331FQ4fPoyhQ4ciOTkZ27dvx4MPPtg5b6SVeJVO7dbN1Rb+LjZIv1KBwykFGNWzcRM3dbGmuh62pVWrvS1gDe93dRgThGtPvKC72G9NGGttgFOpxEBiLHRdKtsTFhtuMlnz4astQe3q2+Z6oW/pFIr6robUuRq23rt0wpAK3QdFugCoC+2taWXqqsd0v+8bBjRra+Nr9TUVDcefdja12jg+rG0gIiLC4P7ixYuxZMkSg315eXlQq9Xw8vIy2O/l5YWEhIQmz3v//fcjLy8P0dHR0Gq1qK2txZNPPomXX365Q+tvKyO6YiFTIwgCokPdsfnvNOxLyjO/0KfR1AeZior6i2RdN7aGt6++39JjHX3fmLseNmxlac8seK3ZVCr+wQfE74EuaPECnIjaSjf5h7U14OYmdTVkbuRyo/vw78yZM/Dz89Pfv7qVr712796NN998E2vWrEFkZCSSkpLw7LPP4rXXXsPChQs75DXag6GPrktUXejbm9RJ6/VptWLAuXpcSWtut+c5DW+bSteW5jTsetielqzraQHTffrKLnRERERkhBwcHODo6NjiMe7u7pDL5cjOzjbYn52dDW9v7yafs3DhQjz44IN49NFHAQB9+/ZFWVkZHn/8cbzyyiuQSXRtxNBH12VEiPhp4NnMYuSVVsHdvhWfktTUAFlZwOXL4paRUX9bt+Xm1oevhl08pCKTiSFGN9mFlZXh7Wvdb8uxbb3fVAgztq6HRERERCZGqVRi0KBBiIuLw4QJEwAAGo0GcXFxmD17dpPPKS8vbxTs5HWtnFoJr2l5VUjXxc1ehXAfR5zNLMb+pDzcGWhrGN6aCnQ5Oe0Lcg3XnWlqjajmbrfnOVffZoAiIiIisjhz587FjBkzMHjwYAwdOhSrV69GWVmZfjbP6dOnw8/PD8uWLQMAxMbGYtWqVRg4cKC+e+fChQsRGxurD39S4JUstU5VFZCZ2TjAZWRg3T/noU2/DN93rwDVla07n0IB+PoCfn71W8P7Xl7ieK2GwcvKimO3iIiIiKjL3HfffcjNzcWiRYuQlZWFAQMGYMeOHfrJXVJTUw1a9hYsWABBELBgwQJcvnwZHh4eiI2NxRtvvCHVWwAACFop2xm7QHp6OgICApCWlgZ/f3+pyzE+Wi2Qn99kmDO4n5fX+nO6uhqGuasDnZ8f4O7O8V5ERERE1KUsNRuwpc9SpaQAX30FfPklcP58656jUjXZOlfl5Y2Zv15Gup0bvlowAYH+nPWLiIiIiMhYMPRZksJCYMsW4IsvgL/+MnzMw6Plljk/P7EFr4nulSoAmrIDSEspwF/ppQx9RERERERGhKHP3NXUAL/+Krbo/fBD/TIEggCMGQNMnw7cfTfg4HBdLxMd6o5DKQXYl5SHB4YFdkDhRERERETUERj6zJFWCxw7Jrboff21uPyBTkSEGPSmTQM6sB9zVJg7Vu48hwPJ+VBrtJDLOOEKEREREZExYOgzJ2lpwMaNYtg7e7Z+v6cnMHWqGPYGDuyUGTD7+TnBQaVAYXkNzmQUo6+/U4e/BhERERERtR1Dn6krKQG+/14Men/8Ub/+nUoFTJgAPPggcMst4nIHnUghl2FYiBt2nsnG3qQ8hj4iIiIiIiPB0GeK1Gpg1y5xnN733wMVFfWPjRwptuhNngw4dW3wig51x84z2diXlIenRoV06WsTEREREVHTGPpMycmTYovepk3iQuk6PXqILXoPPAAEBUlWXlSoOwDg8MUCVNaoYW0ll6wWIiIiIiISMfQZu8xMMeR9+SVw4kT9fldXYMoUsVVv6NBOGafXViEedvB2tEZWcSWOXrqiD4FERERERCQdhj5jVF4ObNsmturt3AloNOJ+KysgNlZs1bvtNkCplLTMqwmCgKhQd/z3WDr2JuUx9BERERERGQGGPmOh0QB79ohBb8sWoLS0/rHhw8UWvXvvFVv4jFh0mBv+eywd+5LypC6FiIiIiIjA0Ce9s2fFrptffSUuuaATHFw/Ti8sTLr62mhEiNi698/lIhSWV8PZ1rhaI4mIiIiILA1DnxRyc8VF07/8EjhypH6/k5PYmjd9OhAVZRTj9NrKy9EaYZ72OJ9TigMX8jG+r4/UJRERERERWTSGvq5SXQ388IMY9H75BaitFfcrFMD48WLQu+MOwNpa2jo7QFSoO87nlGJvUh5DHxERERGRxBj6ukp1NTBzpjhJCwAMHiwGvSlTAA8PSUvraNGh7tiw/yLH9RERERERGQGGvq5ibw/Mng3I5eJYvfBwqSvqNJHdXSGXCbiYX460gnIEuNpKXRIRERERkcVi6OtKy5dLXUGXcLC2woAAZxy9dAX7L+ThPtduUpdERERERGSxZFIXQOZJt0bf3qR8iSshIiIiIrJsDH3UKaLrQt/+pDxoNFqJqyEiIiIislwMfdQpBgQ4w1YpR35ZNRKzS6Quh4iIiIjIYjH0UadQKmSIDHYFAM7iSUREREQkIYY+6jT14/oY+oiIiIiIpMLQR50mOkwMfYeSC1Bdq5G4GiIiIiIiyyRp6Fu2bBmGDBkCBwcHeHp6YsKECUhMTDQ4prKyErNmzYKbmxvs7e0xadIkZGdnS1QxtUVPLwe42ytRUaPG8dQrUpdDRERERGSRJA19e/bswaxZs3Dw4EHs3LkTNTU1uOWWW1BWVqY/5rnnnsP//vc/fPfdd9izZw8yMjIwceJECaum1hIEQd/Fk+P6iIiIiIikIeni7Dt27DC4v2HDBnh6euLo0aMYOXIkioqK8Omnn2LTpk0YM2YMAGD9+vUIDw/HwYMHMWzYMCnKpjaICnXHD/EZ2JuUh7m39JS6HCIiIiIii2NUY/qKiooAAK6u4qyPR48eRU1NDWJiYvTH9OrVC926dcOBAweaPEdVVRWKi4v1W0kJlwuQkq6l70R6EYoraySuhoiIiIjI8hhN6NNoNJgzZw6ioqLQp08fAEBWVhaUSiWcnZ0NjvXy8kJWVlaT51m2bBmcnJz0W0RERGeXTi3wc7ZBsLsd1BotDiUXSF0OEREREZHFMZrQN2vWLJw6dQqbN2++rvPMnz8fRUVF+u3MmTMdVCG1V1SoGwCO6yMiIiIikoJRhL7Zs2fjp59+wh9//AF/f3/9fm9vb1RXV6OwsNDg+OzsbHh7ezd5LpVKBUdHR/3m4ODQmaVTK0RzvT4iIiIiIslIGvq0Wi1mz56NrVu34vfff0dwcLDB44MGDYKVlRXi4uL0+xITE5Gamorhw4d3dbnUTsO7u0MQgKScUmQVVUpdDhERERGRRZF09s5Zs2Zh06ZN+OGHH+Dg4KAfp+fk5AQbGxs4OTnhkUcewdy5c+Hq6gpHR0c8/fTTGD58OGfuNCFOtlbo5+eEE+lF2JeUh0mD/K/9JCIiIiIi6hCStvStXbsWRUVFGDVqFHx8fPTbN998oz/m3XffxR133IFJkyZh5MiR8Pb2xvfffy9h1dQeXK+PiIiIiEgakrb0abXaax5jbW2NDz/8EB9++GEXVESdJTrUHWt2X8DepDxotVoIgiB1SUREREREFsEoJnIh83dDoAtUChlySqqQlFMqdTlERERERBaDoY+6hLWVHEODXQGwiycRERERUVdi6KMuE6VfuiFf4kqIiIiIiCwHQx91Gd16fQeT81Gr1khcDRERERGRZWDooy4T4eMIZ1srlFbV4kR6kdTlEBERERFZBIY+6jIymYCoEC7dQERERETUlRj6qEvVj+tj6CMiIiIi6goMfdSldOP6jqdeQVlVrcTVEBERERGZP4Y+6lLd3GwR4GqDGrUWhy8WSF0OEREREZHZY+ijLqcf13eeXTyJiIiIiDobQx91OY7rIyIiIiLqOgx91OVGhLgBABKySpBbUiVxNURERERE5o2hj7qcm70KET6OAID9F9jaR0RERETUmRj6SBLRYVyvj4iIiIioKzD0kST04/rO50Gr1UpcDRERERGR+WLoI0kMCXKBUi5DRlElLuaXS10OEREREZHZYugjSdgqFbgh0BkAZ/EkIiIiIupMDH0kmei6Lp77GfqIiIiIiDoNQx9JRjeub/+FfKg1HNdHRERERNQZGPpIMn39nOBgrUBRRQ1OZxRJXQ4RERERkVli6CPJKOQyDO8uLtTOcX1ERERERJ2DoY8kxfX6iIiIiIg6F0MfSUo3ru/vi1dQWaOWuBoiIiIiIvPD0EeS6u5uBx8na1TXanDk4hWpyyEiIiIiMjsMfSQpQRAwIkRs7eO4PiIiIiKijsfQR5KLDhMnc+G4PiIiIiKijsfQR5KLqmvpO5VRhCtl1RJXQ0RERERkXhj6SHKejtbo4WUPrRY4kJwvdTlERERERGaFoY+Mgm4WT47rIyIiIiLqWAx9ZBSiQ7leHxERERFRZ2DoI6MQ2d0NcpmAS/nlSCsol7ocIiIiIiKzwdBHRsFepcDAAGcAbO0jIiIiIupIDH1kNDiuj4iIiIiMzYcffoigoCBYW1sjMjIShw8fbvH4wsJCzJo1Cz4+PlCpVOjRowe2b9/eRdU2jaGPjEZ0mBj6DlzIh0ajlbgaIiIiIrJ033zzDebOnYvFixfj2LFj6N+/P8aNG4ecnJwmj6+ursbNN9+MixcvYsuWLUhMTMTHH38MPz+/Lq7ckELSVydqYECAM+yUcuSXVSMhqwQRvo5Sl0REREREFmzVqlV47LHH8NBDDwEA1q1bh59//hmfffYZ5s2b1+j4zz77DAUFBdi/fz+srKwAAEFBQV1ZcpPY0kdGw0ouQ2R3NwAc10dEREREnaekpATFxcX6raqqqtEx1dXVOHr0KGJiYvT7ZDIZYmJicODAgSbP++OPP2L48OGYNWsWvLy80KdPH7z55ptQq9Wd9l5ag6GPjArH9RERERFRZ4uIiICTk5N+W7ZsWaNj8vLyoFar4eXlZbDfy8sLWVlZTZ43OTkZW7ZsgVqtxvbt27Fw4UKsXLkSr7/+eqe8j9Zi904yKrr1+g6nFKCqVg2VQi5xRURERERkbs6cOWMwzk6lUnXIeTUaDTw9PfHRRx9BLpdj0KBBuHz5Mt5++20sXry4Q16jPRj6yKj08LKHu70KeaVVOJ5aiGF13T2JiIiIiDqKg4MDHB1bnj/C3d0dcrkc2dnZBvuzs7Ph7e3d5HN8fHxgZWUFuby+4SI8PBxZWVmorq6GUqm8/uLbgd07yagIgoDoUI7rIyIiIiJpKZVKDBo0CHFxcfp9Go0GcXFxGD58eJPPiYqKQlJSEjQajX7fuXPn4OPjI1ngAxj6yAiN4Lg+IiIiIjICc+fOxccff4zPP/8cZ8+exVNPPYWysjL9bJ7Tp0/H/Pnz9cc/9dRTKCgowLPPPotz587h559/xptvvolZs2ZJ9RYAsHsnGSHdZC4n0gpRXFkDR2sriSsiIiIiIkt03333ITc3F4sWLUJWVhYGDBiAHTt26Cd3SU1NhUxW344WEBCAX3/9Fc899xz69esHPz8/PPvss3jppZekegsAAEGr1Zr1Ktjp6ekICAhAWloa/P39pS6HWmnMO7uRnFeGjx4chFt6N91nmoiIiIioLSw1G7B7JxklXWsfx/UREREREV0fhj4ySlyvj4iIiIioYzD0kVEa3t0NMgG4kFuGzKIKqcshIiIiIjJZDH1klJxsrdDX3xkAsC8pX9piiIiIiIi6QFBQEJYuXYrU1NQOPS9DHxktrtdHRERERJZkzpw5+P7779G9e3fcfPPN2Lx5M6qqqq77vAx9ZLQajusz80lmiYiIiIgwZ84cxMfH4/DhwwgPD8fTTz8NHx8fzJ49G8eOHWv3eRn6yGjd0M0F1lYy5JZUISmnVOpyiIiIiIi6xA033ID33nsPGRkZWLx4MT755BMMGTIEAwYMwGeffdbmBhGGPjJa1lZyDAlyBcBZPImIiIjIctTU1ODbb7/FnXfeieeffx6DBw/GJ598gkmTJuHll1/GtGnT2nQ+RSfVSdQhokPd8df5POxLysNDUcFSl0NERERE1GmOHTuG9evX4+uvv4ZMJsP06dPx7rvvolevXvpj7r77bgwZMqRN52XoI6OmG9d3MLkANWoNrORsnCYiIiIi8zRkyBDcfPPNWLt2LSZMmAArK6tGxwQHB2PKlCltOi9DHxm1CB9HuNha4Up5DU6mF2JQoKvUJRERERERdYrk5GQEBga2eIydnR3Wr1/fpvOy2YSMmkwmYIRuFs/zXK+PiIiIiMxXTk4ODh061Gj/oUOHcOTIkXafl6GPjF50Xejjen1EREREZM5mzZqFtLS0RvsvX76MWbNmtfu8DH1k9KJCxNB3LPUKyqpqJa6GiIiIiKhznDlzBjfccEOj/QMHDsSZM2fafV6GPjJ63dxsEeBqg1qNFodTCqQuh4iIiIioU6hUKmRnZzfan5mZCYWi/dOxMPSRSdB18eR6fURERERkrm655RbMnz8fRUVF+n2FhYV4+eWXcfPNN7f7vAx9ZBKiOK6PiIiIiMzcO++8g7S0NAQGBmL06NEYPXo0goODkZWVhZUrV7b7vFyygUzCiLpxfQlZJcgpqYSng7XEFRERERERdSw/Pz+cPHkSGzduxIkTJ2BjY4OHHnoIU6dObXLNvtZi6COT4GqnRG9fR5zOKMaBC/m4a4Cf1CUREREREXU4Ozs7PP744x16ToY+MhnRoe44nVGMvefzGPqIiIiIyGydOXMGqampqK6uNth/5513tut87Qp9aWlpEAQB/v7+AIDDhw9j06ZNiIiI6PBUSqQTFeqO//yZjH1JedBqtRAEQeqSiIiIiIg6THJyMu6++278888/EAQBWq0WAPTXvWq1ul3nbddELvfffz/++OMPAEBWVhZuvvlmHD58GK+88gqWLl3arkKIrmVIkCuUchkyiipxMb9c6nKIiIiIiDrUs88+i+DgYOTk5MDW1hanT5/Gn3/+icGDB2P37t3tPm+7Qt+pU6cwdOhQAMC3336LPn36YP/+/di4cSM2bNjQ7mKIWmKjlGNQoAsALt1ARERERObnwIEDWLp0Kdzd3SGTySCTyRAdHY1ly5bhmWeeafd52xX6ampqoFKpAAC7du3S9y3t1asXMjMz210M0bVEh9Ut3XCeoY+IiIiIzItarYaDgwMAwN3dHRkZGQCAwMBAJCYmtvu87Qp9vXv3xrp16/DXX39h586duPXWWwEAGRkZcHNza3cxRNeiW69v/4U8qDVaiashIiIiIuo4ffr0wYkTJwAAkZGRWLFiBfbt24elS5eie/fu7T5vu0Lf8uXL8Z///AejRo3C1KlT0b9/fwDAjz/+qO/22Rp//vknYmNj4evrC0EQsG3bNoPHZ86cCUEQDDZdwCTL1NfPCQ7WChRX1uLU5SKpyyEiIiIi6jALFiyARqMBACxduhQpKSm48cYbsX37drz33nvtPm+7Zu8cNWoU8vLyUFxcDBcXF/3+xx9/HLa2tq0+T1lZGfr374+HH34YEydObPKYW2+9FevXr9ff13UrJcsklwkYEeKGX09nY29SHvoHOEtdEhERERFRhxg3bpz+dmhoKBISElBQUAAXF5frmrm+XaGvoqICWq1WH/guXbqErVu3Ijw83KDQaxk/fjzGjx/f4jEqlQre3t7tKZPMVHSoO349nY19SXmYNTpU6nKIiIiIiK5bTU0NbGxsEB8fjz59+uj3u7q6Xve529W986677sIXX3wBACgsLERkZCRWrlyJCRMmYO3atdddVEO7d++Gp6cnevbsiaeeegr5+fktHl9VVYXi4mL9VlJS0qH1kPR04/qOXLyCiur2rVVCRERERGRMrKys0K1bt3avxdeSdoW+Y8eO4cYbbwQAbNmyBV5eXrh06RK++OKL6+prerVbb70VX3zxBeLi4rB8+XLs2bMH48ePb/EbsWzZMjg5Oem3iIiIDquHjEOwux18nKxRrdbgyKUCqcshIiIiIuoQr7zyCl5++WUUFHTsNW67uneWl5frpxL97bffMHHiRMhkMgwbNgyXLl3qsOKmTJmiv923b1/069cPISEh2L17N8aOHdvkc+bPn4+5c+fq71++fJnBz8wIgoCoUHdsOZqOvUl5uDHMQ+qSiIiIiIiu2wcffICkpCT4+voiMDAQdnZ2Bo8fO3asXedtV+gLDQ3Ftm3bcPfdd+PXX3/Fc889BwDIycmBo6Njuwppje7du8Pd3R1JSUnNhj6VSmUw2UtxcXGn1UPSia4Lffu4SDsRERERmYkJEyZ0ynnbFfoWLVqE+++/H8899xzGjBmD4cOHAxBb/QYOHNihBTaUnp6O/Px8+Pj4dNprkGkYESquB3k6oxgFZdVwtVNKXBERERER0fVZvHhxp5y3XaFv8uTJiI6ORmZmpn6NPgAYO3Ys7r777lafp7S0FElJSfr7KSkpiI+Ph6urK1xdXfHqq69i0qRJ8Pb2xoULF/Diiy8iNDS0TTOEknnydLBGTy8HJGaX4MCFfNzejx8EEBERERE1pV2hDwC8vb3h7e2N9PR0AIC/v3+bFmYHgCNHjmD06NH6+7qxeDNmzMDatWtx8uRJfP755ygsLISvry9uueUWvPbaa1yrjwCIs3gmZpdgb1IeQx8RERERmTyZTNbienztndmzXaFPo9Hg9ddfx8qVK1FaWgoAcHBwwPPPP49XXnkFMlnrJgUdNWoUtFpts4//+uuv7SmPLER0mBs+25fCcX1EREREZBa2bt1qcL+mpgbHjx/H559/jldffbXd521X6HvllVfw6aef4q233kJUVBQAYO/evViyZAkqKyvxxhtvtLsgotYaGuwGhUxAakE50grKEeBqK3VJRERERETtdtdddzXaN3nyZPTu3RvffPMNHnnkkXadt13r9H3++ef45JNP8NRTT6Ffv37o168f/vWvf+Hjjz/Ghg0b2lUIUVvZqxQY2M0ZANjaR0RERERma9iwYYiLi2v389sV+goKCtCrV69G+3v16tXhCwkStSQq1B0AsJehj4iIiIjMUEVFBd577z34+fm1+xztCn39+/fHBx980Gj/Bx98gH79+rW7GKK2iq4Lffsv5EOjaX58KBERERGRsXNxcdGvZODq6goXFxc4ODjgs88+w9tvv93u87ZrTN+KFStw++23Y9euXfo1+g4cOIC0tDRs37693cUQtVX/AGfYKeUoKKvG2axi9PZ1krokIiIiIqJ2effddw1m75TJZPDw8EBkZCRcXFzafd52hb6bbroJ586dw4cffoiEhAQAwMSJE/H444/j9ddfx4033tjugojawkouw7DubohLyMG+pDyGPiIiIiIyWTNnzuyU87Z7nT5fX99Gs3SeOHECn376KT766KPrLoyotaJC3RGXkIO9Sfl4fGSI1OUQEREREbXL+vXrYW9vj3vuucdg/3fffYfy8nLMmDGjXedt15g+ImMSHSaO6zucko+q2vYtWElEREREJLVly5bB3d290X5PT0+8+eab7T4vQx+ZvDBPe3g4qFBZo8GxS4VSl0NERERE1C6pqakIDg5utD8wMBCpqantPi9DH5k8QRAQFeIGgOv1EREREZHp8vT0xMmTJxvtP3HiBNzc3Np93jaN6Zs4cWKLjxcWFra7EKLrERXqjm3xGdiblIcXxvWUuhwiIiIiojabOnUqnnnmGTg4OGDkyJEAgD179uDZZ5/FlClT2n3eNoU+J6eWZ0Z0cnLC9OnT210MUXvpFmk/mV6IoooaONlYSVwREREREVHbvPbaa7h48SLGjh0LhUKMahqNBtOnT7+uMX1tCn3r169v9wsRdSZfZxt097BDcm4ZDibnY1xvb6lLIiIiIiJqE6VSiW+++Qavv/464uPjYWNjg759+yIwMPC6ztvuJRuIjE10qDuSc8uwLymPoY+IiIiITFZYWBjCwsI67HycyIXMhq6L515O5kJEREREJmjSpElYvnx5o/0rVqxotHZfWzD0kdkY1t0NMgFIzi1DRmGF1OUQEREREbXJn3/+idtuu63R/vHjx+PPP/9s93kZ+shsONlYoZ+/MwAu3UBEREREpqe0tBRKpbLRfisrKxQXF7f7vAx9ZFai67p47r+QL3ElRERERERt07dvX3zzzTeN9m/evBkRERHtPi8nciGzEhXqjg/+SMLepDxotVoIgiB1SURERERErbJw4UJMnDgRFy5cwJgxYwAAcXFx2LRpE7Zs2dLu8zL0kVm5IdAZ1lYy5JZU4XxOKXp4OUhdEhERERFRq8TGxmLbtm148803sWXLFtjY2KB///74/fff4erq2u7zsnsnmRWVQo6hwW4AgL3nOa6PiIiIiEzL7bffjn379qGsrAzJycm499578cILL6B///7tPidDH5md6FAx9HEyFyIiIiIyRX/++SdmzJgBX19frFy5EmPGjMHBgwfbfT527ySzo1uv72ByPmrUGljJ+dkGERERERm3rKwsbNiwAZ9++imKi4tx7733oqqqCtu2bbuuSVwAtvSRGQr3doSrnRJl1WqcSCuUuhwiIiIiohbFxsaiZ8+eOHnyJFavXo2MjAy8//77HXZ+hj4yOzKZgOEhdeP62MWTiIiIiIzcL7/8gkceeQSvvvoqbr/9dsjl8g49P0MfmSXden0c10dERERExm7v3r0oKSnBoEGDEBkZiQ8++AB5eR13HcvQR2ZJF/qOpxaitKpW4mqIiIiIiJo3bNgwfPzxx8jMzMQTTzyBzZs3w9fXFxqNBjt37kRJScl1nZ+hj8xSgKsturnaolajxeGUfKnLISIiIiK6Jjs7Ozz88MPYu3cv/vnnHzz//PN466234OnpiTvvvLPd52XoI7Olm8Vz73mGPiIiIiIyLT179sSKFSuQnp6Or7/++rrOxdBHZovj+oiIiIjI1MnlckyYMAE//vhju8/B0Edma3iIGwQBSMwuQU5JpdTlEBEREZEJ+vDDDxEUFARra2tERkbi8OHDrXre5s2bIQgCJkyY0LkFtgJDH5ktVzslevs6AgD2J7GLJxERERG1zTfffIO5c+di8eLFOHbsGPr3749x48YhJyenxeddvHgRL7zwAm688cYuqrRlDH1k1vTj+tjFk4iIiIjaaNWqVXjsscfw0EMPISIiAuvWrYOtrS0+++yzZp+jVqsxbdo0vPrqq+jevXsXVts8hj4ya7pxffuT8qDVaiWuhoiIiIiMQUlJCYqLi/VbVVVVo2Oqq6tx9OhRxMTE6PfJZDLExMTgwIEDzZ576dKl8PT0xCOPPNIptbcHQx+ZtSFBrlAqZMgoqkRKXpnU5RARERGREYiIiICTk5N+W7ZsWaNj8vLyoFar4eXlZbDfy8sLWVlZTZ537969+PTTT/Hxxx93St3tpZC6AKLOZG0lx+BAF+y/kI99SXno7mEvdUlEREREJLEzZ87Az89Pf1+lUl33OUtKSvDggw/i448/hru7+3WfryMx9JHZiwp1x/4L+diblIcHhwdJXQ4RERERSczBwQGOjo4tHuPu7g65XI7s7GyD/dnZ2fD29m50/IULF3Dx4kXExsbq92k0GgCAQqFAYmIiQkJCOqD6tmP3TjJ7+nF9F/Kh1nBcHxERERFdm1KpxKBBgxAXF6ffp9FoEBcXh+HDhzc6vlevXvjnn38QHx+v3+68806MHj0a8fHxCAgI6MryDbClj8xeHz8nOForUFxZi38uF2FAgLPUJRERERGRCZg7dy5mzJiBwYMHY+jQoVi9ejXKysrw0EMPAQCmT58OPz8/LFu2DNbW1ujTp4/B852dnQGg0f6uxtBHZk8uEzAixB07TmdhX1IeQx8RERERtcp9992H3NxcLFq0CFlZWRgwYAB27Nihn9wlNTUVMpnxd54UtGY+j316ejoCAgKQlpYGf39/qcshiXx54CIW/nAaw7u74evHh0ldDhERERFJwFKzgfHHUqIOoFuk/eilK6ioVktcDRERERFR12HoI4sQ7G4HXydrVKs1+PtigdTlEBERERF1GYY+sgiCIOhb+/Yl5UlcDRERERFR12HoI4sRHSaGvr0MfURERERkQRj6yGKMCBFD3+mMYhSUVUtcDRERERFR12DoI4vh4aBCL28HAMD+C2ztIyIiIiLLwNBHFoXj+oiIiIjI0jD0kUWJDuW4PiIiIiKyLAx9ZFGGBrtCIROQVlCB1PxyqcshIiIiIup0DH1kUexUCtzQzQUAsI/j+oiIiIjIAjD0kcWJYhdPIiIiIrIgDH1kcaLD3AAA+5PyoNFoJa6GiIiIiKhzMfSRxenn7wx7lQJXymtwJrNY6nKIiIiIiDoVQx9ZHCu5DMO6uwLg0g1EREREZP4Y+sgijQjhuD4iIiIisgwMfWSRosPE0Pf3xQJU1qglroaIiIiIqPMw9JFFCvO0h4eDCpU1GhxLvSJ1OUREREREnYahjyySIAiIrlu6geP6iIiIiMicMfSRxapfry9f4kqIiIiIiDoPQx9ZrKhQcb2+f9ILUVReI3E1RERERESdg6GPLJaPkw1CPOyg0QIHktnaR0RERETmiaGPLBrH9RERERGRuWPoI4sWxdBHRERERGaOoY8s2rAQN8gEIDmvDBmFFVKXQ0RERETU4Rj6yKI5Wluhf4AzALb2EREREZF5Yugji8dxfURERERkzhj6yOI1XK9Pq9VKXA0RERERUceSNPT9+eefiI2Nha+vLwRBwLZt2wwe12q1WLRoEXx8fGBjY4OYmBicP39emmI7QnW51BVQEwZ2c4aNlRx5pVU4l10qdTlERERERB1K0tBXVlaG/v3748MPP2zy8RUrVuC9997DunXrcOjQIdjZ2WHcuHGorKzs4ko7yBd3Ah8MAX55CTj3G1BdJnVFBEClkGNosCsAYC+7eBIRERGRmVFI+eLjx4/H+PHjm3xMq9Vi9erVWLBgAe666y4AwBdffAEvLy9s27YNU6ZM6cpSr19VKZB5AlBXA3nngEPrALkS6DYMCBkLhI4FvPoAgiB1pRYpOtQde87lYl9SHh6JDpa6HCIiIiKiDiNp6GtJSkoKsrKyEBMTo9/n5OSEyMhIHDhwoNnQV1VVhaqqKv39kpKSTq+1VVT2wAvngZQ9wIXfgaTfgaJUIOVPcdu1GLDzBELG1G/2HlJXbTFGhLoBAA4m56NGrYGVnMNdiYiIiMg8GG3oy8rKAgB4eXkZ7Pfy8tI/1pRly5bh1Vdf7dTa2s3GGYi4S9y0WiD/AnAhDkiKAy7+BZTlACc3ixsAePcTw1/oWCAgElCoJC3fnIV7O8LVTomCsmrEpxViSJCr1CUREREREXUIow197TV//nzMnTtXf//y5cuIiIiQsKJmCALgHipukU8AtVVA2iExAF74Hcg6Wb/tWw1Y2QFB0WIADBkDuIWyK2gHkskEjAhxw08nM7H3fB5DHxERERGZDaMNfd7e3gCA7Oxs+Pj46PdnZ2djwIABzT5PpVJBpapvESsuLu60GjuUQgUEjxS3m18FSnOA5N31IbAsBzj/q7gBgFM3IGS0GAKDbxJbEem6RIe646eTmdiXlIfnbu4hdTlERERERB3CaENfcHAwvL29ERcXpw95xcXFOHToEJ566ilpi+sK9p5Av3vFTaMBck7XBcA4IPWgOB7w2OfiJsgAv8H1rYC+NwByo/2nNVq69fqOpxWipLIGDtZWEldERERERHT9JE0GpaWlSEpK0t9PSUlBfHw8XF1d0a1bN8yZMwevv/46wsLCEBwcjIULF8LX1xcTJkyQrmgpyGSAd19xi54jLvVwcZ8YAC/8Ls4Gmn5Y3HYvA6ydgO6j6iaEGQs4B0j9DkxCgKstAt1scSm/HIdTCjA23OvaTyIiIiIiMnKShr4jR45g9OjR+vu6sXgzZszAhg0b8OKLL6KsrAyPP/44CgsLER0djR07dsDa2lqqko2D0g7ocYu4AUBhqhj+LvwudgmtLALO/CBuAODeoz4ABkWJz6cmRYW641J+KvYm5TH0EREREZFZELRarVbqIjpTeno6AgICkJaWBn9/f6nL6XzqWiDjeP2soJePAFpN/eMN1wYMGSOuDSjj8gQ62//JxL82HkMPL3v89txNUpdDRERERB3I4rJBHQ78MjdyBRAwRNxGzQMqCq+9NmD3m8QxgX43iF1IrWykfheSGd7dDYIAnMsuRU5xJTwdLbxVmYiIiIhMHkOfuWu0NmBSXQBssDbgP9+JGwDIFIBnhBgAfW8A/AYBHr0sZmIYFzsl+vg64Z/LRdh3IQ93D7ScT4DMUv4F4OyPwKX94rqXA6cBrt2lroqIiIioS1nGlTyJBAFwDxM33dqAqQfF7fJRIOMYUJZbvz7g0Q3i8xQ2gE9/MQj6DQJ8B4oXzma6TmBUqLsY+pLyGfpMUd554Mw2cUxr1j/1+8//Bvz1DhB0IzDwASD8TkBpK1mZRERERF2Foc+SKVRi187udWPXtFqgKL0+AF4+BmTEA9UlQNpBcdOxdm7QGlgXBh28pXgXHS461B3r9lzAvqQ8aLVaCGYabs2GVgvkJtRPXpRzpv4xQS6ufdl9lNil+cLvYgv3xb+A7f8H9JkIDHxQ/PnlvzMRERGZKU7kQi3TaID883UBsC4IZp0E1NWNj3XwrQuCA+tbBE1p0Xh1DVCShaor6Xj+s1/hrsnHs0Pt4VKbB5RkiktlePcFAiLFzS2EQUEqWi2Qfao+6OWdq39MZiWGvIi7gF63A7au9Y8VpgEnNgPHvwQKL9Xv9wgXW//63QfYe3TZ2yAiIjJ7Wq3YoKCwBrz7SF2NxWYDhj5qu9pqcbH4hkEwN8FwllAd1xAxAOpaBX36df1EMVotUHFFDG7FmUBJRt3Xuq04Q/xalgegDf8dbFyBgKF1W6T4/thdsPNotUBmfH3QK0iuf0yuFGekjbgL6HkrYOPS8rk0GuDSXuD4V+K5aivF/TIF0ONWsfUvNMZixrKiME1sBU09ADj6AX0mAV4RUldFpk6jFj9Ms+KEWEQWqSAZOPkdcPIboOAC0Ptu4J4NUldlsdmAoY86RlWp2AJ4+Vh999ArFxsfJ8jFi8mG3UI9wtt/cV1TeVV4yzIMciWZ4j7dRf21yKwABx9kal1w7IoKKhd/xEQOABx8xGCRcQxIOyy+T3XVVc9ViEtgBETWB0Enf7YGXg/dp4O6MXqFqfWPKazFYBYxAegxDrB2bN9rVBQCp78Hjn0p/vvq2HsDA6YCAx4A3EOv400YoapS4OLe+vU98883PsYzQuz+2nui2KpN1BpVJeJEYYnbgXO/ApWFgFO3uvHkPQCPHuJX9x6AnQd/PxKZm/IC4NR/gZPfAumH6/db2YofKN75vuT/7y01GzD0UecpL6hvCdSFwbKcxscpbMQWQN1soX43AC5BQHl+C61zdfsqrrS+HhtXwNFXDHCOPuJXB5/6fQ4+gK0bIJPhZHoh7vxgHxysFTi+8GYo5FetZVhbLYbctMNA2iFxK8ls/JoOvnVLaNR1CfXuByiUbfo2WhyNRvxDceYH4MyPQHF6/WNWtkDYLWKLXtgtgMq+Y187+zRwfCNwcrP486fTbbjY/TNiQse/ZlfQaMRW0gu/Axf+EH9eNTX1jwtywH8wEBQN5CQASTsNu3D7DhT/WPeeCDj5dXn5ZOSKLgPnfgEStovjZZvq/t8Ua6f6ANhwcwmynFZ2Mm21VeIH3C7Blv23vaZS/B1w8ltx0jRNrbhfkInDLfrdJw63UDlIWqaOpWYDhj7qOlotUHzZsFtoxnGgqriJgwW0uqulwrpBgGsY5LzF0OXoI7batKGLkVqjxcClv6G4shZb/zUCA7tdo7ugbhKc9MP1QTDzJKBVGx4nV4kX0LqWwIChgL1nq+syWxq12LVQF/RKs+ofU9qLXS4j7hJb9rqiC21tNXBuh9j9M2lnfddlpb3YPWXgg+K/nTG3UhRdrm/JS94NVBQYPu4cCISOBULGiDOaNhx/W1EIJPwkflqbvMfw57jbCLEFMGICxz9aKq1WnBk38RexRS8z3vBx1+5Az9vEizy3UHHplLxEcWbdvHPiduUSmv0dL7MSz+EeBnj0rAuDYYBbWPtb9Ik6Qm21eP2SUjchWNphoLZC/NsQPFL8fRoaA7gGS11p59NogEv7xK6bZ34wvJbz7icGvb6TjXKSP0vNBgx9JC2NRuznrQ+CR8WwpK4CIIjdfxx9xPDm4N10S52NS6dcfD/55VHsOJ2FF27pgdljwtp+gupyMdSmHaoPgldfeAPip9oNu4R6RgAy+XXXb/TUteK4ujM/AGf/Jy4XoqNyFC8aI+4S/4hKOSaoOAM48bUYABuOI3QLE1v/+k8FHLykq0+nugy4uK8+6OUlGj6udBBn6g0ZLX5PW7teYWkucPYH4J//Aqn76/cLMiD4JvGPeq87TGvSpuulUQMQAJnsmoeajdpq8QIvcbsY9orSGjwoiL+/eo4Het4uBrRr/U6uqRD/P+WdA3LP1YfB/CSgprz55zn41HUVbRAG3XuIfxuM+UMYMk3qGvHv+MW/xKCXdqjxz6dc2bh127W7GP5CxgLBNwJKu66rubPlnBWD3snvDHviOPoD/e4Rw55nuHT1tYKlZgOGPjI+6hpxUhU7d0BuJVkZXx68hIXbTmFYd1dsfnz49Z9QqxU/8U4/XB8Ec86i0afdSnuxm6uuS6j/YPO5oFbXACl76oLeT4Yh2NpZDA8Rd4nhRKGSrMwmabVia+Txr4DTW+v/8AtysavpDQ+KX7vqZ1ajAbL/qQ95qQcNLzwEmfhzFDJG3PwGXX9tRZfF935qi3ghpCNXihc4fSaJrbKm2AW2JcWZQPrfddsR8b0LMnE2X5/+gO8AwGeAGD7MqVtiRSGQtAtI+Fn82vCTfIWN+HPVc7w4prajeixoNGKPkLxzdS2DDVoIS7Obf57SXmxVvHrcoGt34/tdQsZLXQtkngAu/imOe049CFSXGh5j6yZ2hQ+6UWzdcwsTZ5NO2iWOZ007WN+9ERB/P3YbLvasCI0RP9g1tQ8oSrKAf7aIQx8arn+rcgJ63yUGvW4jTOaDMEvNBgx9RM1IySvD6Hd2QymXIX7xzbBVdsLFXGWReBGpawlMPyKui3g1j14NuoRGihc3pvJHo7ZK7F545gfx4rGysP4xW7f6oBc8UtKQ3yaVxWL4Of6V4UB1Ow+g/xSx+6dHz45/3eJMIPmP+rF55XmGjzt1A0LrQl7wyGvPYno98i+IE+Cc+t5wbUQrWzH49ZkkXuCY2syNNZXiRV/DkNfw0+yWKGzE6ch9BtSHQY9epvNzDYjdLnXdNi/tM7x4tfMQ/2173S628nb1bMUVhWJLoK5VUNdCWJDcuCu9jiATe1O496xvFXTuJra8WNnUbbbiV4WN+G9lKr9b6fpp1HUhb6/YmnfpQOO/wTYu9SEv6Ebx/3RL4aayWDxXUpwYBBsuDwSIrdUhY8Xf1d1HGy4rZEyqSsVu/ic2ix/W6oY5yKzEDzj73Sv+PjC13/Gw3GzA0EfUDK1Wi+jlf+ByYQU+f3goburRBeOXNGpx+Qt9l9DDYvfXq9m4AP5DxUlvVI7ihYpc2cLXutuyZvY3fLwjPqmrqRD/4J39UbyAbNhCYOcJhMeKQS8wyvRbRnITxfB34mvDLqr+Q8Tun70ntn8cUnW52KXyQl3QaxiuALF1I+jG+tY8qdaOzD4jjv879V/gSkr9fpWj+G/dZ6IYEowt/Gi14gVZ+pH6kJd50nCSG0AMDl69xX9T/yGA32AAWiAjXhzPlnlC3K5uEQDEcbxevetaA/uLgdAzwngmfdBN8qPrtpl9yvBxj1513TZvE9+3MX6SX1stTqaRd+6q7XwzY8ZbIMjrQ6DB1mCf4up9tuKF79X7FNZXnUt3nK34O5fhsuvpekik/CUGvUv7gaoiw2OsnYDAaLFbZlA04Nm7/T/3Wq34oUTSLnFL+UscA6ij75FR1wrod4O0wzvUteKHtCc3ix/SNuzKGhApBr3eE403qLaSpWYDhj6iFry45QS+PZKOx0d2x8u3SdRHvSyvwSyhh8Wxj61dgqI9ZIr2BUbd7eoy8Y9GwwtgBx8g/E4x6HUbZp5jFtU14qxlx78Sp6rXtTxY2YqTngx8AAgc0fKFnm7ReV2XzUsHrloaRBAnAtKFPP8hxhMeALH+jONi+Du9Veymp2PrJv7795kkXTegqlLx/4+uBS/9b8OgrmPnIX6o4j9Y/B77Drx2l1Xd+OTME+L3IPOEGCCvvqAExP9LXhGGLYKevbvuE/OaSrElQhf0Gs48LMjErmg9bxPDnikv16HVil1CG4bAvHNiN+XaCvHDqZoK8cK2qXVmO5MgM2xhtHUVA0boWPH/hwm2nhgljUb8sEw3Ju/SPsPeJoD44VTgiLrumjeKSy911t+omkpxmMCFOPGD0as/yLN2rht3PVb8WXD07Zw6GtKtgXviG7HrfsPfia4hYu+VvpNbPw7cBFhqNmDoI2rBD/GX8ezmeET4OGL7szdKXY6otlr8pDLtsBgQairFsVzqGrGFQndbXX3V7dom9lWjTQvSt4Wjv3iRH3GXeOFsjC0EnaUkW/yk9NiXhmvguXYHBkwDBtxf/8e8JNuwy+bVy5o4+tWHvO6jTOcTVo1GHNty6r/A6W2GXVEdfMRPi/tMEj/Z7owWD41G7Aqo76b5t3iBdfXFvcxKDF7+Q+pDnnO3jqlJoxFbPjNP1LcIZsQ3vugExA9bPMIB37rWQJ8BYldRK5vrrwMQl9A596sY9C78bvihjJWdeIHZ63ax25ap/Ix1FK1W/J1YU14fAmsr628bfL0qLNZUNAiQ5Vc9Vtl4X3PdUBtSWIu9IHSz63r0Yqtga2m14lj5i3vrxuXtazyBmtIBCBxe32XTp790H0TqZllO2iX+Hai86kMiz4j6sYDdhnfs+NQrl4B/vhWXWcg7V7/f1g3oM1kcp9dZv58lZqnZgKGPqAW5JVUY8sYuAMDRBTFwszfDCQE06qbDoLqmwf0mQqSmpun9Wq14wWKmfyzaRKsVw/nxL8WWL92FtiATx9yV5TXuTmdla9hlszUzIRo7da04JuTU9+JMrQ1bv5wDxfDXd/L1TXBQcQVIP1of8C4faXwBBQBOAfXhzn+IOLV4V7aq6LqU6gJgZrz4tamZfQW5ODbUZ0B991Dvvq2fCTD/Qn1rXuoBw8Dr4FPfbTPoRrYsdQVduGwqJF65WN/Cf/War45+9bPuGvMYMClotWJgSambeOXi3sZjna3sxB4mwXVj8nwGGOewAnWt2BNBNxbw8lEYfCir+9ugC4Gu3dv++7Liivgh3MlvDWdjVliLH/r0u0/8OTO2rvgdzFKzAUMf0TXcuvpPJGSV4P2pAxHbvwu6WpB5qioVJ7M5/pXhH1tAvAjRhbyAoeY922BtlXhRc+q/YiBpOGbEo5cYAPtMarlboboWyD0rhru0upDXsEVVR2EjfvigC3l+g8XlXoyNbp1PXYugLgw21fVUkImzBepmDPXpD/j0Exc91mjEsJu4XVwo/eplO7z61Hfb9BlgWa3vpkLXUnUhrq6L9/6ruvPXdfHWtQL6DzH7C3QDWq3Ygq/rrnlxb+PeEQobMeQFRYsfrvkONM3vUXmB2PqXVNcVtOH6tUDdWqsx4hZ8Y/MLn9dWiUMPTn4jtvbrZ3kWxO9Pv/vEsdcWtAampWYDhj6ia3j9pzP4ZG8KpgwJwFuT+kldDpmDvCTxwtzRV+yyaecudUXSqC4Dzu0QWwDP/2a45IRPf7GLUe+7xRDccDbNy8eAmrLG53MNMeym6dXbNC/2APHitiTTsEUw80TjViAAgCDO6FtZaBgUZQqx1V0X9FwCu6Z26jg1FWLw07UCXj0GTOVYtyh43Tgwc1oUvLpMnCgrN0EMwrkJ4jjZq8OPwlr8sCxopBj0/AYZ11jnjqDVAtmn68YC7hLHezeccEpmJQZd3eLwXn3EeQBOfiP2MmnYpdyrjzghS5/JgJNfl78VY2Cp2YChj+ga/kjMwUPr/0aAqw3+enGM1OUQmafKInHtxlP/FScCutbYJ6UD4D+obsKVIeKFnp1bl5QqqZJsw/GBmScMl5RQOQJhN4tBLzTGfNb4JFFxRt1svnHi16u7BbsE17cCBo9svvXHmFSX1y3B0SDc5ZwFClPR5JhzuVL8f6/rruk3yPK6J1eViq2cSbvEn4WCZMPHFdaGLcQOvuLC6X3vFccKWzhLzQYMfUTXUFZViwFLf0ONWovfnhuJHl4m8EeUyJSV5YldYU/9V2zlAADPcMOxeO49zHMW2PYozQWyTogXwwHDzK+Vg5qmW25D1wqYdshwXUWZQpxmX9cKKHWX3poKcQbVq8PdlYtodkIxW3fx/75HL8Czlzju13dgx01wZC4Kkuu7gab8KfaEUDqIE6n1u1dsAeXvSz1LzQYMfUStcO9/DuBwSgFkAnBDNxeMCfdETLgXwjztIZj6JBtExqy8QLx4taDxJkTtUlUijnO78HvTrT82rvUTwoSM6bzlAGoqxTG2OQni2Fvd1ysXm18aw8a1Qbhr8NVSu75fj9oqseXUNQRQ2kpdjVGy1GzA0EfUCgeT87H0f2dwJtNwoV9/FxuM7eWJseFeiOzuCpWCn6QREZERKEipbwVM+bPxQvUe4fVdQQNHtL31rLa6LtydNWy9K0huIdy5iK/r2cvwq5276c9STCbDUrMBQx9RG2QUVuD3hBzEnc3Gvgv5qK6t/8Nmp5TjxjAPjAn3xJhennA3x+UdiIjI9KhrxEmQdK2Al4/BoEulwloMfiFjxK6gnuH1Iay2Gii40Djc5V9ofuyttVPT4c7ek+GOJGep2YChj6idyqtrsS8pH78nZCPubA5ySqr0jwkC0N/fGTHhnhjTywvhPg7sBkpERMahvECcMEnXElh82fBxBx9xBt0rF8UlEhqOFWxI5VQX6HoahjsHb4Y7MlqWmg0Y+og6gEajxemMYuw6m43fE3Lwz2XDRaH9nG0wppcnxoR7Ynh3N1hbsRsoEREZAd0C50l1awNe3CsuIN+Q0qEu0F015s7Bh+GOTI6lZgOGPqJOkFVUiT8SxW6ge5PyUFlT3w3UxkqO6DB3jO0ldgP1dLSwqaaJiMh41VQCaQfFmTZdgsWw5+jHcEdmw1KzAUMfUSerrFFj/4U8xJ3NQdzZHGQVVxo83s/fCWN7eWFsuCd6+zqyGygRERFRJ7HUbMDQR9SFtFotzmQWiwEwIQcn0goNHvd2tMaYcE+M7eWJESHusFGyGygRERFRR7HUbCDhKp1ElkcQBPT2dcIzY8Pww6woHH5lLFZM6odbIrxgq5Qjq7gSmw6l4pHPj2Dga7/hkQ1/Y+OhS8gqqrz2yYmIiIiow3344YcICgqCtbU1IiMjcfjw4WaP/fjjj3HjjTfCxcUFLi4uiImJafH4rsKWPiIjUVmjxsHk/LolIXJwudBwIH1vX0eMDffC2F6e6OvnBJmM3UCJiIiI2qKt2eCbb77B9OnTsW7dOkRGRmL16tX47rvvkJiYCE9Pz0bHT5s2DVFRURgxYgSsra2xfPlybN26FadPn4afn19nvKVWYegjMkJarRaJ2SV14wCzcTytEA3/p3o4qDCmpyfGhnsiOswdtkqFdMUSERERmYi2ZoPIyEgMGTIEH3zwAQBAo9EgICAATz/9NObNm3fN56vVari4uOCDDz7A9OnTr7v+9uKVIpEREgQBvbwd0cvbEbNGhyKvtAq7E3MRdzYbf57LRW5JFb45koZvjqRBqZBhRIibOBtouBf8nG2kLp+IiIjIqJWUlKC4uFh/X6VSQaVSGRxTXV2No0ePYv78+fp9MpkMMTExOHDgQKtep7y8HDU1NXB1de2YwtuJoY/IBLjbqzB5kD8mD/JHVa0ah1MK6iaDyUZaQQV2J+Zid2IuFv5wGr28HRAT7oUx4Z4Y4O/MbqBEREREV4mIiDC4v3jxYixZssRgX15eHtRqNby8vAz2e3l5ISEhoVWv89JLL8HX1xcxMTHXVe/1Yuiro1arUVNTI3UZZMSUSiVkMunnPlIp5LgxzAM3hnlgcWwEknJKsetsDn5PyMbRS1eQkFWChKwSfPBHEtztlRjV0xMx4Z6IDvOAvYr/5YmIiIjOnDljMMbu6la+jvDWW29h8+bN2L17N6ytpV2X2eKvALVaLbKyslBYWCh1KWTkZDIZgoODoVQqpS5FTxAEhHk5IMzLAU+NCkFBWTX2nMvBrrM5+DMxF3ml1dhyNB1bjqZDKZchsrsrxvbyxNhwLwS42kpdPhEREZEkHBwc4Ojo2OIx7u7ukMvlyM7ONtifnZ0Nb2/vFp/7zjvv4K233sKuXbvQr1+/6673ell86NMFPk9PT9ja2nJhbGqSRqNBRkYGMjMz0a1bN6P9OXG1U+Lugf64e6A/atQa/J1SgLgEcTKYi/nl+Ot8Hv46n4cl/zuDHl72+tlAB3ZzgZzdQImIiIj0lEolBg0ahLi4OEyYMAGAeE0YFxeH2bNnN/u8FStW4I033sCvv/6KwYMHd1G1LbPo0KdWq/WBz83NTepyyMh5eHggIyMDtbW1sLKykrqca7KSyzAi1B0jQt2x4PZwJOeVIe5sNuLO5uDIpSs4l12Kc9mlWLv7AlxsrTC6p9gCeGMPdzhaG//7IyIiIupsc+fOxYwZMzB48GAMHToUq1evRllZGR566CEAwPTp0+Hn54dly5YBAJYvX45FixZh06ZNCAoKQlZWFgDA3t4e9vb2kr0Piw59ujF8trbs5kbXpuvWqVarTSL0NSQIAkI87BHiYY/HR4agsLwae87lIu5sDnYn5uBKeQ2+P34Z3x+/DIVMQGR3V4zp5YWYcE8EutlJXT4RERGRJO677z7k5uZi0aJFyMrKwoABA7Bjxw795C6pqakGcz6sXbsW1dXVmDx5ssF5mpoopitZ9Dp9lZWVSElJQXBwsOSDK8n4mevPS61ag6OXrui7gV7ILTN4PMTDTpwNtJcnBgW6QCGXfjIbIiIiovaw1DW8Lbqlj4gAhVyGyO5uiOzuhpdvC8fFvDLsOpuN3xNycDilABdyy3AhNxn/+TMZTjZWGNXTA2PDvXBTmAecbE2rxZOIiIjIEjH0EZGBIHc7PHpjdzx6Y3cUVdTgr/NiN9A/EnNQWF6DH+Iz8EN8BuQyAUOCXDC2lxfGhnuiu4d0/dSJiIiIqHkMfYSgoCDMmTMHc+bMue5z7d69G6NHj8aVK1fg7Ox83ecjaTnZWOGOfr64o58v1Botjqde0a8JeC67FAeTC3AwuQBvbD+LYHc7jO3liTG9PNE/wBl2XBOQiIiIyCjwqsxEjRo1CgMGDMDq1auv+1x///037Ow4WQe1TC4TMDjIFYODXDFvfC+k5pfj94RsxCXk4GByPlLyyvDJ3hR8sjcFABDgaoOeXo7o5e2Ant4OCPdxQJCbHccEEhEREXUxhj4zpdVqoVaroVBc+5/Yw8OjCyoic9PNzRYzo4IxMyoYpVW1+OtcLuIScvDX+VxkF1chraACaQUV2HW2fkFTpUKGUA97fRDs6e2AXt6O8HJUGe3ah0RERESmjh+5N6DValFeXSvJ1pZJVGfOnIk9e/bg3//+NwRBgCAI2LBhAwRBwC+//IJBgwZBpVJh7969uHDhAu666y54eXnB3t4eQ4YMwa5duwzOFxQUZNBiKAgCPvnkE9x9992wtbVFWFgYfvzxx3Z/X//73/+id+/eUKlUCAoKwsqVKw0eX7NmDcLCwmBtbQ0vLy+DKW63bNmCvn37wsbGBm5uboiJiUFZWdnVL0ESs1cpML6vD965pz8OvRyDYwtvxqbHIrE4NgJThgRgQIAzbJVyVNdqcCazGN8fv4xlvyRg5vq/MWxZHAa+thP3/ecAFv9wCpsOpeLopSsoraqV+m0RERERmQW29DVQUaNGxKJfJXntM0vHwVbZun+Of//73zh37hz69OmDpUuXAgBOnz4NAJg3bx7eeecddO/eHS4uLkhLS8Ntt92GN954AyqVCl988QViY2ORmJiIbt26Nfsar776KlasWIG3334b77//PqZNm4ZLly7B1dW1Te/r6NGjuPfee7FkyRLcd9992L9/P/71r3/Bzc0NM2fOxJEjR/DMM8/gyy+/xIgRI1BQUIC//voLAJCZmYmpU6dixYoVuPvuu1FSUoK//vqrTQGZpOFqp8SIEHeMCHHX79NotEi/UoGzWcVIzCpBYlYJErKKkZJXhsLyGhxKKcChlAKD8/i72KCXd30X0V7eDgh2ZxdRIiIiorZg6DNBTk5OUCqVsLW1hbe3NwAgISEBALB06VLcfPPN+mNdXV3Rv39//f3XXnsNW7duxY8//ojZs2c3+xozZ87E1KlTAQBvvvkm3nvvPRw+fBi33nprm2pdtWoVxo4di4ULFwIAevTogTNnzuDtt9/GzJkzkZqaCjs7O9xxxx1wcHBAYGAgBg4cCEAMfbW1tZg4cSICAwMBAH379m3T65PxkMkEdHOzRTc3W4zr7a3fX1mjRlJOqT4EJtQFwpySKqRfqUD6lau6iMplCPEUu4j2YhdRIiIiomti6GvAxkqOM0vHSfbaHWHw4MEG90tLS7FkyRL8/PPP+hBVUVGB1NTUFs/Tr18//W07Ozs4OjoiJyenzfWcPXsWd911l8G+qKgorF69Gmq1GjfffDMCAwPRvXt33Hrrrbj11lv13Ur79++PsWPHom/fvhg3bhxuueUWTJ48GS4uLm2ug4yXtZUcffyc0MfPyWD/lbLqugBYjMTsEpzNLMG57BKUV6txNrMYZzOLDY53srHStwbqgmBPbwfYcxZRIiIisnC8GmpAEIRWd7E0VlfPwvnCCy9g586deOeddxAaGgobGxtMnjwZ1dXVLZ7Hyspw0W1BEKDRaDq8XgcHBxw7dgy7d+/Gb7/9hkWLFmHJkiX4+++/4ezsjJ07d2L//v347bff8P777+OVV17BoUOHEBwc3OG1kHFxsVNieIgbhoe46ffpuogm1HURTcgWWwVT8spQVFGDwykFONxkF1HdxDGO+i6iVuwiSkRERBbCtBOOBVMqlVCr1dc8bt++fZg5cybuvvtuAGLL38WLFzu5unrh4eHYt29fo5p69OgBuVxs3VQoFIiJiUFMTAwWL14MZ2dn/P7775g4cSIEQUBUVBSioqKwaNEiBAYGYuvWrZg7d26XvQcyHg27iN7STBfRxOwSfQthdnHDLqL1LdUNu4j2bDBe0NvRml1EiYiIyOww9JmooKAgHDp0CBcvXoS9vX2zrXBhYWH4/vvvERsbC0EQsHDhwk5psWvO888/jyFDhuC1117DfffdhwMHDuCDDz7AmjVrAAA//fQTkpOTMXLkSLi4uGD79u3QaDTo2bMnDh06hLi4ONxyyy3w9PTEoUOHkJubi/Dw8C6rn0xDa7uIJmSV4FxWCcpa3UXUAT28HOBgbdjyTURERGRKGPpM1AsvvIAZM2YgIiICFRUVWL9+fZPHrVq1Cg8//DBGjBgBd3d3vPTSSyguLm7y2M5www034Ntvv8WiRYvw2muvwcfHB0uXLsXMmTMBAM7Ozvj++++xZMkSVFZWIiwsDF9//TV69+6Ns2fP4s8//8Tq1atRXFyMwMBArFy5EuPHj++y+sm0NddF9HJhBRKySpCQWdyqLqJ+zjYI92EXUSIiIjJNgtbM579PT09HQEAA0tLS4O/vb/BYZWUlUlJSEBwcDGtra4kqJFPBnxfzVlmjxoXc0gbLSYiziWYXVzV5vFIuQ3cPO3EWUR9HdhElIiIyAS1lA3PGlj4iIohdRHv7OqG3r2EX0cLyav0yEvquonVdRBPq9iE+Q3+8o7VCP3NoT28HhPuwiygRERFJi6GP2uTJJ5/EV1991eRjDzzwANatW9fFFRF1LmdbJYZ1d8Ow7k13EU1ssLZgcl4ZiitrcfhiAQ5fbNxF1HDiGEcEutnCuoOWayEiIiJqDkMftcnSpUvxwgsvNPmYo6NjF1dDJA2ZTECAqy0CXG1xc4SXfn9VrRoXcsqQmF2MhMwSfRjMKq7E5cIKXC6sQFyC4XqXLrZW8HGyga+zDXydretui199nKzh7WTNsYNERER0XRj6qE08PT3h6ekpdRlERkmlkCPC1xERvo7AwPr9heXVBstJJGQW43x2KUqqanGlvAZXymtwJrPpCZYEAfB0UDUKg2JItIGvkzXc7VWQyTiOkIiIiJrG0EdE1MmcbZWI7O6GyAZdRLVaLYorapFRVIHMogpkFFYis6gCmYViq2BmUSWyiipRrdYgu7gK2cVViE9r+vxWcgFejtbwdbKBT10w9NMFRGdxv7OtFSeYISIislAMfUREEhAEAU62VnCytUK4T9NdozUaLfLLqvWhMKOwLiAWVSKzLhhmF1eiRq3VL0LfHBsrub6F0MfJGj51rYQNv9qr+CeBiIjIHPEvPBGRkZLJBHg4qODhoEK/ZmaVrlVrkFNShYxCwzCYUfc1s6gCeaXVqKhRIzmvDMl5Zc2+nqO1onEorGst9HO2gbeTNVQKTjxDRERkahj6iIhMmEIu04/va05ljRpZRZViV9K6bqQZumBYKO4vqaxFcWUtinXLUDTD3V5pMK6wfnyhGBA9HVRQcOIZIiIio8LQR0Rk5qyt5Ahyt0OQu12zx5RW1SKzQWuhvuWwqD4YVtZokFdajbzSavxzuajJ88hlQt3EM/WTzfg4Gc5K6m6v5PhCIiKiLsTQZ6GCgoIwZ84czJkz55rHCoKArVu3YsKECZ1eFxFJw16lQJiXA8K8HJp8XKvVorC8Rj/JTFOTz2QXV6JWo617vBLHUgubPJdSLqubcMZw8hlfZ13roQ0crRUMhkRERB2EoY+IiK5JEAS42CnhYqdEHz+nJo9Ra7TIK63SjyfMKKwPhroWxNzSKlSrNbiUX45L+eXNvp6dUg6fulZCP93yFHXdSDm+kIiIqG0Y+oiIqEPIZeLSEV6O1g2XKTRQXatBdnFlfShsOM6w7uuV8hqUVauRlFOKpJzSJs8jCICHvQq+zjZ1obC+O6kuJLpwmQoiIiIADH2GtFqgpvlPnjuVla14FdMKH330EZYsWYL09HTIZPUTJtx1111wc3PDK6+8grlz5+LgwYMoKytDeHg4li1bhpiYmA4p9Z9//sGzzz6LAwcOwNbWFpMmTcKqVatgb28PANi9ezdefPFFnD59GlZWVujduzc2bdqEwMBAnDhxAnPmzMGRI0cgCALCwsLwn//8B4MHD+6Q2ojIuCkVMgS42iLA1bbZYyqq1fowqP9aFxAvX6nA5cIKVNWKs5bmlFQhPq2wyfNYW8nqQ6CTYUuhr7PYrZSthUREZAkY+hqqKQfe9JXmtV/OAJTNT7LQ0D333IOnn34af/zxB8aOHQsAKCgowI4dO7B9+3aUlpbitttuwxtvvAGVSoUvvvgCsbGxSExMRLdu3a6rzLKyMowbNw7Dhw/H33//jZycHDz66KOYPXs2NmzYgNraWkyYMAGPPfYYvv76a1RXV+Pw4cP6T9unTZuGgQMHYu3atZDL5YiPj4eVldV11URE5sVGKUeIhz1CPOybfFyr1aKgrBoZdWMJM/QTz1Tgcl1AzC2pQmWNBsm5ZUjObX6ZCg8HXWuhdYNgWN966GrHSWeIiMj0MfSZIBcXF4wfPx6bNm3Sh74tW7bA3d0do0ePhkwmQ//+/fXHv/baa9i6dSt+/PFHzJ49+7pee9OmTaisrMQXX3wBOzsxpH7wwQeIjY3F8uXLYWVlhaKiItxxxx0ICQkBAISHh+ufn5qaiv/7v/9Dr169AABhYWHXVQ8RWR5BEOBmr4KbvQp9/ZseX1hVKy5TcbmwfmH7jEKxlVAXFCtrNMgtqUJuSRVOpDX9WiqFrMGYwsZdSH2crGFtxdZCIiIybkYd+pYsWYJXX33VYF/Pnj2RkJDQOS9oZSu2uEnBqvmuTk2ZNm0aHnvsMaxZswYqlQobN27ElClTIJPJUFpaiiVLluDnn39GZmYmamtrUVFRgdTU1Osu8+zZs+jfv78+8AFAVFQUNBoNEhMTMXLkSMycORPjxo3DzTffjJiYGNx7773w8fEBAMydOxePPvoovvzyS8TExOCee+7Rh0Mioo6iUsgR6GaHQLeme1BotVpcKa/RB0F9a2GD1sOckipU1Wquuai9u71KbCl0bthSWH/fja2FREQkMaMOfQDQu3dv7Nq1S39foejEkgWh1V0spRYbGwutVouff/4ZQ4YMwV9//YV3330XAPDCCy9g586deOeddxAaGgobGxtMnjwZ1dXVXVLb+vXr8cwzz2DHjh345ptvsGDBAuzcuRPDhg3DkiVLcP/99+Pnn3/GL7/8gsWLF2Pz5s24++67u6Q2IiJAbC10tVPCtYXZSKtq1cguqtKHwKu/ZhRWoqJGjbzSKuSVVuFEetNrF6oUsvqWQifDlkJd6yFbC4mIqDMZfehTKBTw9vaWugyjY21tjYkTJ2Ljxo1ISkpCz549ccMNNwAA9u3bh5kzZ+qDVGlpKS5evNghrxseHo4NGzagrKxM39q3b98+yGQy9OzZU3/cwIEDMXDgQMyfPx/Dhw/Hpk2bMGzYMABAjx490KNHDzz33HOYOnUq1q9fz9BHREZHpZCjm5sturk13ROj4dqF9eMKDccZ6loLU/LKkNJia6FSDIFNTDjj68wF7YmI6PoYfeg7f/48fH19YW1tjeHDh2PZsmUtTkZSVVWFqqoq/f2SkpKuKFMS06ZNwx133IHTp0/jgQce0O8PCwvD999/j9jYWAiCgIULF0Kj0XTYay5evBgzZszAkiVLkJubi6effhoPPvggvLy8kJKSgo8++gh33nknfH19kZiYiPPnz2P69OmoqKjA//3f/2Hy5MkIDg5Geno6/v77b0yaNKlDaiMi6kqtWbtQt0RF+pWmJ5y5fKWirrWwGnml1TjZTGuhUiGDr1PTXUh14ZCthURE1ByjDn2RkZHYsGEDevbsiczMTLz66qu48cYbcerUKTg4ODT5nGXLljUaB2iuxowZA1dXVyQmJuL+++/X71+1ahUefvhhjBgxAu7u7njppZdQXFzcIa9pa2uLX3/9Fc8++yyGDBlisGSD7vGEhAR8/vnnyM/Ph4+PD2bNmoUnnngCtbW1yM/Px/Tp05GdnQ13d3dMnDjRYv69iMjyXGuJCq1Wi6KKmiYnnNF1Ic0uqUR1rQYX88txsYUF7Z1treBmp4SbvQru9kq426vgZqeCW91td3tl3QQ4SjioFGw5JCKyIIJWq9VKXURrFRYWIjAwEKtWrcIjjzzS5DFXt/RdvnwZERERSEtLg7+/v8GxlZWVSElJQXBwMKytrTu1djJ9/HkhIinoWgsbdhu9fFVALK9Wt+mcSoUM7nb1IdBd99Xuqvv2KrjaKWEll137pEREJiA9PR0BAQFNZgNzZtQtfVdzdnZGjx49kJSU1OwxKpUKKpVKf7+jWriIiIik0NrWwpwScUKZ/NJq5JdWIa+0GvlldV9L67+WVatRXatBRlElMooqW1VDw1ZEj7pAyFZEIiLTYVKhr7S0FBcuXMCDDz4odSlmY+PGjXjiiSeafCwwMBCnT5/u4oqIiKgtBEGAs60SzrZK9PBqeuhDQxXVaoMwmF9ajVxdWCwTv+bVhcSCsipotEBheQ0Ky2twoYWF7nUatiI2DIPudiq4OxiGRbYiEhF1DaMOfS+88AJiY2MRGBiIjIwMLF68GHK5HFOnTpW6NLNx5513IjIyssnHrKysurgaIiLqbDZKOfyVtvB3ufb6sBqNFoUVNfplKaRsRXS3V8KerYhERO1i1KEvPT0dU6dORX5+Pjw8PBAdHY2DBw/Cw8ND6tLMhoODQ7OT4hARkWWTyerXM2xvK2JeWRXyStiKSEQkJaMOfZs3b5a6BCIiImolU2lF9HCom+HUQdzv4aDikhdEZNaMOvQRERGReTK2VkQHlQLudWFQDIXiVn+7fj8DIhGZGoY+IiIiMnrtaUXML62qn6SmtAr5ZdXIrZvlNLe0Gnkl4uPVtRqUVNWipKoWKXmtCIjWCngYhEKlYUB0qN+vUjAgEpH0GPqIiIjIrDRsRQy7RiuiVqtFSVWtGAZLxJbC3JJK5NW1HOpCori/CtVqDUoqa1FSWYvkVgRER2tdC6IYBD3sm25NdGNAJKJOxNBHREREFksQBDhaW8HR2gohHvYtHqvValFcWWsYButaC/NKxKUvGu6rUYvHF1fWIrkVXUydbKwatRrqg6JD/X43OxWUCk5SQ0Stx9BnoYKCgjBnzhzMmTNH6lKIiIhMgiAIcLKxgpNNKwNiRS1yDVoLG7cc6vbXqLUoqqhBUUXrxiA621o1GGtobRAWPRqERjd7zmJKRAx9RERERB1OEAQ42VrBydYKoZ7XDohFdTOZ5tR1Ma1vQdSNQRRbE/NKq1Cr0eonqUnKuXYtLvqA2HDcobKuBbE+JHKZCyLzxdBHJketVkMQBMhk/MNERESmTxAEONsq4WyrRKhny2MQNZr6gJirC4ZXtRrqbueXVqNWo8WV8hpcKa/B+ZzSa9biaqeEu704HlKlkEOpkEGpkEEll+lvKxvebni/mWNUChmUcnmTz1HV3ZbJhI76dhJRExj6GtJqgfJyaV7b1hYQWvcL76OPPsKSJUuQnp5uEHzuuusuuLm54ZVXXsHcuXNx8OBBlJWVITw8HMuWLUNMTEy7Slu1ahXWr1+P5ORkuLq6IjY2FitWrIC9ff0nl/v27cMrr7yCw4cPQ6VSYejQodi8eTNcXFyg0Wjwzjvv4KOPPkJaWhq8vLzwxBNP4JVXXsHu3bsxevRoXLlyBc7OzgCA+Ph4DBw4ECkpKQgKCsKGDRswZ84cfPHFF5g3bx7OnTuHpKQk5Obm4uWXX8bx48dRU1ODAQMG4N1338UNN9ygr6uwsBAvvfQStm3bhqKiIoSGhuKtt97C6NGj4ePjg88++wyTJ0/WH79t2zZMmzYNWVlZXLSeiIiMjkwmwMVOCZdWTFLTcC1E/aylJYathvqAWFYNtUaLgrJqFJRVd9G7qaeQCU2HSnldMGz0mNwwOF4zdLYtkFrJBQitvC4jMgUMfQ2VlwP2LXfB6DSlpYCdXasOveeee/D000/jjz/+wNixYwEABQUF2LFjB7Zv347S0lLcdttteOONN6BSqfDFF18gNjYWiYmJ6NatW5tLk8lkeO+99xAcHIzk5GT861//wosvvog1a9YAEEPa2LFj8fDDD+Pf//43FAoF/vjjD6jVagDA/Pnz8fHHH+Pdd99FdHQ0MjMzkZCQ0KYaysvLsXz5cnzyySdwc3ODp6cnkpOTMWPGDLz//vvQarVYuXIlbrvtNpw/fx4ODg7QaDQYP348SkpK8NVXXyEkJARnzpyBXC6HnZ0dpkyZgvXr1xuEPt19Bj4iIjJ1bVkLUaPR4kp5tX7W0oKyalTXalCt1ohf625X1Ta8rzZ4rLq2weNXPa+6wfOq6u43VKvRorZajfJqdWd+S9qkUQtnM+HRxkoOG6UcNlZyWDe4bWMlh3WD2zZKmfh4M8eway11JoY+E+Ti4oLx48dj06ZN+tC3ZcsWuLu7Y/To0ZDJZOjfv7/++Ndeew1bt27Fjz/+iNmzZ7f59RpO9hIUFITXX38dTz75pD70rVixAoMHD9bfB4DevXsDAEpKSvDvf/8bH3zwAWbMmAEACAkJQXR0dJtqqKmpwZo1awze15gxYwyO+eijj+Ds7Iw9e/bgjjvuwK5du3D48GGcPXsWPXr0AAB0795df/yjjz6KESNGIDMzEz4+PsjJycH27duxa9euNtVGRERk6mQyAW72KrjZq9ATnf/Bp1arRY1a2ygQVqvVVwXLlkJn49uGoVPd6LiqFgJprUZrUKNuP6o6/dsBQGztvDooirfrg6U+NF59/6rQadtMCFUp2JXWUjH0NWRrK7a4SfXabTBt2jQ89thjWLNmDVQqFTZu3IgpU6ZAJpOhtLQUS5Yswc8//4zMzEzU1taioqICqamp7Spt165dWLZsGRISElBcXIza2lpUVlaivLwctra2iI+Pxz333NPkc8+ePYuqqip9OG0vpVKJfv36GezLzs7GggULsHv3buTk5ECtVqO8vFz/PuPj4+Hv768PfFcbOnQoevfujc8//xzz5s3DV199hcDAQIwcOfK6aiUiIqKWCYIApULs0gmV1NWI1BotatTNtFY2E0irajSorFWjolqNyho1KmrUqKjWoKKm7n513b6r7je8rcuatRpxzciSqtpOfZ/WuhDZqCWypVApa0VLZv3x7B5rfBj6GhKEVnexlFpsbCy0Wi1+/vlnDBkyBH/99RfeffddAMALL7yAnTt34p133kFoaChsbGwwefJkVFe3vY/+xYsXcccdd+Cpp57CG2+8AVdXV+zduxePPPIIqqurYWtrCxsbm2af39JjAPRjErXa+k/XampqmjzP1b88ZsyYgfz8fPz73/9GYGAgVCoVhg8frn+f13ptQGzt+/DDDzFv3jysX78eDz30EH9JERERWSC5TIBcJgaXrqLViq2dlXVBsaKZYNjkff1tzVWhs8HtuvtVDbrTVtZoUFmjwRU0vt7qKHJdq2WDwDgkyBVv3N23016TWsbQZ6Ksra0xceJEbNy4EUlJSejZs6d+ApN9+/Zh5syZuPvuuwEApaWluHjxYrte5+jRo9BoNFi5cqU+oH377bcGx/Tr1w9xcXF49dVXGz0/LCwMNjY2iIuLw6OPPtrocQ8PDwBAZmYmXFxcAIgtdK2xb98+rFmzBrfddhsAIC0tDXl5eQZ1paen49y5c8229j3wwAN48cUX8d577+HMmTP6LqhEREREnU0QBKgUcqgUcjjBqtNeR63Roqr26kBpGDQrmwid5dVXB8ymQqWm7thafaulWqNFaVUtShu0Wvo5X/vDeOo8DH0mbNq0abjjjjtw+vRpPPDAA/r9YWFh+P777xEbGwtBELBw4UJoNJoWztS80NBQ1NTU4P3330dsbCz27duHdevWGRwzf/589O3bF//617/w5JNPQqlU4o8//sA999wDd3d3vPTSS3jxxRehVCoRFRWF3NxcnD59Go888ghCQ0MREBCAJUuW4I033sC5c+ewcuXKVtUWFhaGL7/8EoMHD0ZxcTH+7//+z6B176abbsLIkSMxadIkrFq1CqGhoUhISIAgCLj11lsBiOMjJ06ciP/7v//DLbfcAn9//3Z9n4iIiIiMlVwmwFapgK2y8y79deM0m2uVdLDuvFBL18ZpgkzYmDFj4OrqisTERNx///36/atWrYKLiwtGjBiB2NhYjBs3zmAZg7bo378/Vq1aheXLl6NPnz7YuHEjli1bZnBMjx498Ntvv+HEiRMYOnQohg8fjh9++AEKhfiLZeHChXj++eexaNEihIeH47777kNOjriarJWVFb7++mskJCSgX79+WL58OV5//fVW1fbpp5/iypUruOGGG/Dggw/imWeegaenp8Ex//3vfzFkyBBMnToVERERePHFF/Wziurouqo+/PDD7foeEREREVk6cZymDE42VvBytEaQux3CfRxxQzcXjAh1R19/J6lLtGiCtuFgKjOUnp6OgIAApKWlNWrFqaysREpKCoKDg2FtbS1RhSS1L7/8Es899xwyMjKgVCqbPY4/L0RERESmraVsYM7YvZMsVnl5OTIzM/HWW2/hiSeeaDHwERERERGZKnbvtHAbN26Evb19k5turT1ztWLFCvTq1Qve3t6YP3++1OUQEREREXUKtvRZuDvvvBORkZFNPmZlZd4DbpcsWYIlS5ZIXQYRERERUadi6LNwDg4OcHBwkLoMIiIiIiLqJOzeCcOFwYmaw58TIiIiIjJFFh36dN0Xy8vLJa6ETEF1dTUAQC6XS1wJEREREVHrWXT3TrlcDmdnZ/2acba2thAEQeKqyBhpNBrk5ubC1tZWv/4gEREREZEpsPirV29vbwDQBz+i5shkMnTr1o0fDBARERGRSbH40CcIAnx8fODp6YmamhqpyyEjplQqIZNZdI9oIiIiIjJBFh/6dORyOcdqERERERGR2WGzBRERERERUTM+/PBDBAUFwdraGpGRkTh8+HCLx3/33Xfo1asXrK2t0bdvX2zfvr2LKm0eQx8REREREVETvvnmG8ydOxeLFy/GsWPH0L9/f4wbN67Z+UD279+PqVOn4pFHHsHx48cxYcIETJgwAadOneriyg0JWjNffCw9PR0BAQFIS0uDv7+/1OUQEREREZFE2poNIiMjMWTIEHzwwQcAxBndAwIC8PTTT2PevHmNjr/vvvtQVlaGn376Sb9v2LBhGDBgANatW9dxb6SNzH5Mn0ajAQBkZmZKXAkREREREUlJlwmKiorg6Oio369SqaBSqQyOra6uxtGjRzF//nz9PplMhpiYGBw4cKDJ8x84cABz58412Ddu3Dhs27atg95B+5h96MvOzgYADB06VOJKiIiIiIjIGPTp08fg/uLFi7FkyRKDfXl5eVCr1fDy8jLY7+XlhYSEhCbPm5WV1eTxWVlZ11/0dTD70Ddw4EAcPnwYXl5enG6f2q2kpAQRERE4c+YMHBwcpC6HzAB/pqgj8eeJOhJ/nqijGdPPlEajQWpqKiIiIqBQ1Eehq1v5zI3Zhz6FQoEhQ4ZIXQaZuOLiYgCAn5+fQVcAovbizxR1JP48UUfizxN1NGP7merWrVurjnN3d4dcLtf3HNTJzs6Gt7d3k8/x9vZu0/FdhU1fREREREREV1EqlRg0aBDi4uL0+zQaDeLi4jB8+PAmnzN8+HCD4wFg586dzR7fVcy+pY+IiIiIiKg95s6dixkzZmDw4MEYOnQoVq9ejbKyMjz00EMAgOnTp8PPzw/Lli0DADz77LO46aabsHLlStx+++3YvHkzjhw5go8++kjKt8HQR9QaKpUKixcvNvv+3tR1+DNFHYk/T9SR+PNEHc2Uf6buu+8+5ObmYtGiRcjKysKAAQOwY8cO/WQtqampBvOGjBgxAps2bcKCBQvw8ssvIywsDNu2bWs0cUxXM/t1+oiIiIiIiCwZx/QRERERERGZMYY+IiIiIiIiM8bQR0REREREZMYY+oiIiIiIiMwYQx9RC5YtW4YhQ4bAwcEBnp6emDBhAhITE6Uui8zEW2+9BUEQMGfOHKlLIRN2+fJlPPDAA3Bzc4ONjQ369u2LI0eOSF0WmSC1Wo2FCxciODgYNjY2CAkJwWuvvQbO+Uet8eeffyI2Nha+vr4QBAHbtm0zeFyr1WLRokXw8fGBjY0NYmJicP78eWmKtUAMfUQt2LNnD2bNmoWDBw9i586dqKmpwS233IKysjKpSyMT9/fff+M///kP+vXrJ3UpZMKuXLmCqKgoWFlZ4ZdffsGZM2ewcuVKuLi4SF0amaDly5dj7dq1+OCDD3D27FksX74cK1aswPvvvy91aWQCysrK0L9/f3z44YdNPr5ixQq89957WLduHQ4dOgQ7OzuMGzcOlZWVXVypZeKSDURtkJubC09PT+zZswcjR46UuhwyUaWlpbjhhhuwZs0avP766xgwYABWr14tdVlkgubNm4d9+/bhr7/+kroUMgN33HEHvLy88Omnn+r3TZo0CTY2Nvjqq68krIxMjSAI2Lp1KyZMmABAbOXz9fXF888/jxdeeAEAUFRUBC8vL2zYsAFTpkyRsFrLwJY+ojYoKioCALi6ukpcCZmyWbNm4fbbb0dMTIzUpZCJ+/HHHzF48GDcc8898PT0xMCBA/Hxxx9LXRaZqBEjRiAuLg7nzp0DAJw4cQJ79+7F+PHjJa6MTF1KSgqysrIM/u45OTkhMjISBw4ckLAyy6GQugAiU6HRaDBnzhxERUWhT58+UpdDJmrz5s04duwY/v77b6lLITOQnJyMtWvXYu7cuXj55Zfx999/45lnnoFSqcSMGTOkLo9MzLx581BcXIxevXpBLpdDrVbjjTfewLRp06QujUxcVlYWAMDLy8tgv5eXl/4x6lwMfUStNGvWLJw6dQp79+6VuhQyUWlpaXj22Wexc+dOWFtbS10OmQGNRoPBgwfjzTffBAAMHDgQp06dwrp16xj6qM2+/fZbbNy4EZs2bULv3r0RHx+POXPmwNfXlz9PRCaO3TuJWmH27Nn46aef8Mcff8Df31/qcshEHT16FDk5ObjhhhugUCigUCiwZ88evPfee1AoFFCr1VKXSCbGx8cHERERBvvCw8ORmpoqUUVkyv7v//4P8+bNw5QpU9C3b188+OCDeO6557Bs2TKpSyMT5+3tDQDIzs422J+dna1/jDoXQx9RC7RaLWbPno2tW7fi999/R3BwsNQlkQkbO3Ys/vnnH8THx+u3wYMHY9q0aYiPj4dcLpe6RDIxUVFRjZaROXfuHAIDAyWqiExZeXk5ZDLDS0O5XA6NRiNRRWQugoOD4e3tjbi4OP2+4uJiHDp0CMOHD5ewMsvB7p1ELZg1axY2bdqEH374AQ4ODvp+505OTrCxsZG4OjI1Dg4OjcaD2tnZwc3NjeNEqV2ee+45jBgxAm+++SbuvfdeHD58GB999BE++ugjqUsjExQbG4s33ngD3bp1Q+/evXH8+HGsWrUKDz/8sNSlkQkoLS1FUlKS/n5KSgri4+Ph6uqKbt26Yc6cOXj99dcRFhaG4OBgLFy4EL6+vvoZPqlzcckGohYIgtDk/vXr12PmzJldWwyZpVGjRnHJBrouP/30E+bPn4/z588jODgYc+fOxWOPPSZ1WWSCSkpKsHDhQmzduhU5OTnw9fXF1KlTsWjRIiiVSqnLIyO3e/dujB49utH+GTNmYMOGDdBqtVi8eDE++ugjFBYWIjo6GmvWrEGPHj0kqNbyMPQRERERERGZMY7pIyIiIiIiMmMMfURERERERGaMoY+IiIiIiMiMMfQRERERERGZMYY+IiIiIiIiM8bQR0REREREZMYY+oiIiIiIiMwYQx8REREREZEZY+gjIiJqgSAI2LZtm9RlEBERtRtDHxERGa2ZM2dCEIRG26233ip1aURERCZDIXUBRERELbn11luxfv16g30qlUqiaoiIiEwPW/qIiMioqVQqeHt7G2wuLi4AxK6Xa9euxfjx42FjY4Pu3btjy5YtBs//559/MGbMGNjY2MDNzQ2PP/44SktLDY757LPP0Lt3b6hUKvj4+GD27NkGj+fl5eHuu++Gra0twsLC8OOPP3bumyYiIupADH1ERGTSFi5ciEmTJuHEiROYNm0apkyZgrNnzwIAysrKMG7cOLi4uODvv//Gd999h127dhmEurVr12LWrFl4/PHH8c8//+DHH39EaGiowWu8+uqruPfee3Hy5EncdtttmDZtGgoKCrr0fRIREbWXoNVqtVIXQURE1JSZM2fiq6++grW1tcH+l19+GS+//DIEQcCTTz6JtWvX6h8bNmwYbrjhBqxZswYff/wxXnrpJaSlpcHOzg4AsH37dsTGxiIjIwNeXl7w8/PDQw89hNdff73JGgRBwIIFC/Daa68BEIOkvb09fvnlF44tJCIik8AxfUREZNRGjx5tEOoAwNXVVX97+PDhBo8NHz4c8fHxAICzZ8+if//++sAHAFFRUdBoNEhMTIQgCMjIyMDYsWNbrKFfv37623Z2dnB0dEROTk573xIREVGXYugjIiKjZmdn16i7ZUexsbFp1XFWVlYG9wVBgEaj6YySiIiIOhzH9BERkUk7ePBgo/vh4eEAgPDwcJw4cQJlZWX6x/ft2weZTIaePXvCwcEBQUFBiIuL69KaiYiIuhJb+oiIyKhVVVUhKyvLYJ9CoYC7uzsA4LvvvsPgwYMRHR2NjRs34vDhw/j0008BANOmTcPixYsxY8YMLFmyBLm5uXj66afx4IMPwsvLCwCwZMkSPPnkk/D09MT48eNRUlKCffv24emnn+7aN0pERNRJGPqIiMio7dixAz4+Pgb7evbsiYSEBADizJqbN2/Gv/71L/j4+ODrr79GREQEAMDW1ha//vornn32WQwZMgS2traYNGkSVq1apT/XjBkzUFlZiXfffRcvvPAC3N3dMXny5K57g0RERJ2Ms3cSEZHJEgQBW7duxYQJE6QuhYiIyGhxTB8REREREZEZY+gjIiIiIiIyYxzTR0REJosjFIiIiK6NLX1ERERERERmjKGPiIiIiIjIjDH0ERERERERmTGGPiIiIiIiIjPG0EdERERERGTGGPqIiIiIiIjMGEMfERERERGRGWPoIyIiIiIiMmP/D2AtuBS1RnhCAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1000x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get loss, val_loss, and the computed metric from history\n",
    "loss = [x['loss'] for x in history if 'loss' in x]\n",
    "val_loss = [x['eval_loss'] for x in history if 'eval_loss' in x]\n",
    "\n",
    "# Get accuracy value \n",
    "metric = [x['eval_spearmanr'] for x in history if 'eval_spearmanr' in x]\n",
    "\n",
    "epochs_loss = [x['epoch'] for x in history if 'loss' in x]\n",
    "epochs_eval = [x['epoch'] for x in history if 'eval_loss' in x]\n",
    "\n",
    "# Create a figure with two y-axes\n",
    "fig, ax1 = plt.subplots(figsize=(10, 5))\n",
    "ax2 = ax1.twinx()\n",
    "\n",
    "# Plot loss and val_loss on the first y-axis\n",
    "# For the loss we plot a horizontal line because we have just one loss value (after the first epoch)\n",
    "# Exchange the two lines below if you trained multiple epochs\n",
    "#line1 = ax1.plot([0]+epochs_loss, loss*2, label='train_loss')\n",
    "line1 = ax1.plot(epochs_loss, loss, label='train_loss')\n",
    "\n",
    "line2 = ax1.plot(epochs_eval, val_loss, label='val_loss')\n",
    "ax1.set_xlabel('Epoch')\n",
    "ax1.set_ylabel('Loss')\n",
    "\n",
    "# Plot the computed metric on the second y-axis\n",
    "line3 = ax2.plot(epochs_eval, metric, color='red', label='val_accuracy')\n",
    "ax2.set_ylabel('Accuracy')\n",
    "ax2.set_ylim([0, 1])\n",
    "\n",
    "# Combine the lines from both y-axes and create a single legend\n",
    "lines = line1 + line2 + line3\n",
    "labels = [line.get_label() for line in lines]\n",
    "ax1.legend(lines, labels, loc='lower left')\n",
    "\n",
    "# Show the plot\n",
    "plt.title(\"Training History\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7a4a53e",
   "metadata": {},
   "source": [
    "# Save and Load the finetuned model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6ade4a9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_model(model,filepath):\n",
    "# Saves all parameters that were changed during finetuning\n",
    "\n",
    "    # Create a dictionary to hold the non-frozen parameters\n",
    "    non_frozen_params = {}\n",
    "\n",
    "    # Iterate through all the model parameters\n",
    "    for param_name, param in model.named_parameters():\n",
    "        # If the parameter has requires_grad=True, add it to the dictionary\n",
    "        if param.requires_grad:\n",
    "            non_frozen_params[param_name] = param\n",
    "\n",
    "    # Save only the finetuned parameters \n",
    "    torch.save(non_frozen_params, filepath)\n",
    "\n",
    "    \n",
    "def load_model(filepath, num_labels=1):\n",
    "# Creates a new PT5 model and loads the finetuned weights from a file\n",
    "\n",
    "    # load a new model\n",
    "    model, tokenizer = PT5_classification_model(num_labels=num_labels)\n",
    "    \n",
    "    # Load the non-frozen parameters from the saved file\n",
    "    non_frozen_params = torch.load(filepath)\n",
    "\n",
    "    # Assign the non-frozen parameters to the corresponding parameters of the model\n",
    "    for param_name, param in model.named_parameters():\n",
    "        if param_name in non_frozen_params:\n",
    "            param.data = non_frozen_params[param_name].data\n",
    "\n",
    "    return tokenizer, model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be5ba621",
   "metadata": {},
   "source": [
    "This saves only the finetuned weights to a .pth file\n",
    "\n",
    "It is a 10 MB File, while the entire model would be around 4.8 GB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d31dc7e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "save_model(model,\"./PT5_disorder_finetuned.pth\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f8e96aa",
   "metadata": {},
   "source": [
    "To load the weights again, we initialize a new PT5 model from the pretrained checkpoint and load the LoRA weights afterwards\n",
    "\n",
    "You need to specifiy the correct num_labels here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "edbd69f7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loading configuration file config.json from cache at /data2/cache/huggingface/hub/models--Rostlab--prot_t5_xl_uniref50/snapshots/973be27c52ee6474de9c945952a8008aeb2a1a73/config.json\n",
      "Model config T5Config {\n",
      "  \"architectures\": [\n",
      "    \"T5ForConditionalGeneration\"\n",
      "  ],\n",
      "  \"d_ff\": 16384,\n",
      "  \"d_kv\": 128,\n",
      "  \"d_model\": 1024,\n",
      "  \"decoder_start_token_id\": 0,\n",
      "  \"dense_act_fn\": \"relu\",\n",
      "  \"dropout_rate\": 0.1,\n",
      "  \"eos_token_id\": 1,\n",
      "  \"feed_forward_proj\": \"relu\",\n",
      "  \"initializer_factor\": 1.0,\n",
      "  \"is_encoder_decoder\": true,\n",
      "  \"is_gated_act\": false,\n",
      "  \"layer_norm_epsilon\": 1e-06,\n",
      "  \"model_type\": \"t5\",\n",
      "  \"n_positions\": 512,\n",
      "  \"num_decoder_layers\": 24,\n",
      "  \"num_heads\": 32,\n",
      "  \"num_layers\": 24,\n",
      "  \"output_past\": true,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"relative_attention_max_distance\": 128,\n",
      "  \"relative_attention_num_buckets\": 32,\n",
      "  \"transformers_version\": \"4.26.1\",\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 128\n",
      "}\n",
      "\n",
      "loading weights file pytorch_model.bin from cache at /data2/cache/huggingface/hub/models--Rostlab--prot_t5_xl_uniref50/snapshots/973be27c52ee6474de9c945952a8008aeb2a1a73/pytorch_model.bin\n",
      "Some weights of the model checkpoint at Rostlab/prot_t5_xl_uniref50 were not used when initializing T5EncoderModel: ['decoder.block.9.layer.1.EncDecAttention.k.weight', 'decoder.block.19.layer.1.EncDecAttention.q.weight', 'decoder.block.10.layer.0.SelfAttention.k.weight', 'decoder.block.9.layer.1.EncDecAttention.v.weight', 'decoder.block.6.layer.0.SelfAttention.k.weight', 'decoder.block.14.layer.0.SelfAttention.k.weight', 'decoder.block.6.layer.0.SelfAttention.o.weight', 'decoder.block.11.layer.2.DenseReluDense.wo.weight', 'decoder.block.13.layer.1.EncDecAttention.v.weight', 'decoder.block.20.layer.1.EncDecAttention.k.weight', 'decoder.block.2.layer.1.EncDecAttention.v.weight', 'decoder.block.0.layer.1.EncDecAttention.v.weight', 'decoder.block.15.layer.0.SelfAttention.o.weight', 'decoder.block.19.layer.0.SelfAttention.k.weight', 'decoder.block.14.layer.2.DenseReluDense.wi.weight', 'decoder.block.22.layer.0.layer_norm.weight', 'decoder.block.5.layer.2.layer_norm.weight', 'decoder.block.1.layer.0.SelfAttention.q.weight', 'decoder.block.11.layer.1.EncDecAttention.q.weight', 'decoder.block.12.layer.0.SelfAttention.o.weight', 'decoder.block.16.layer.1.layer_norm.weight', 'decoder.block.2.layer.1.EncDecAttention.k.weight', 'decoder.block.21.layer.1.EncDecAttention.k.weight', 'decoder.block.23.layer.0.SelfAttention.k.weight', 'decoder.block.9.layer.1.layer_norm.weight', 'decoder.block.3.layer.2.DenseReluDense.wi.weight', 'decoder.embed_tokens.weight', 'decoder.block.21.layer.1.EncDecAttention.v.weight', 'decoder.block.7.layer.1.EncDecAttention.k.weight', 'decoder.block.23.layer.2.DenseReluDense.wo.weight', 'decoder.block.9.layer.0.SelfAttention.o.weight', 'decoder.block.5.layer.0.SelfAttention.v.weight', 'decoder.block.17.layer.2.DenseReluDense.wo.weight', 'decoder.block.16.layer.0.SelfAttention.v.weight', 'decoder.block.11.layer.2.DenseReluDense.wi.weight', 'decoder.block.5.layer.0.SelfAttention.k.weight', 'decoder.block.4.layer.0.layer_norm.weight', 'decoder.block.5.layer.1.EncDecAttention.o.weight', 'decoder.block.16.layer.1.EncDecAttention.k.weight', 'decoder.block.5.layer.1.EncDecAttention.v.weight', 'decoder.block.1.layer.2.layer_norm.weight', 'decoder.block.13.layer.1.EncDecAttention.o.weight', 'decoder.block.15.layer.2.DenseReluDense.wi.weight', 'decoder.block.8.layer.0.SelfAttention.o.weight', 'decoder.block.0.layer.0.SelfAttention.o.weight', 'decoder.block.13.layer.0.layer_norm.weight', 'decoder.block.4.layer.0.SelfAttention.k.weight', 'decoder.block.18.layer.0.SelfAttention.k.weight', 'decoder.block.17.layer.1.EncDecAttention.o.weight', 'decoder.block.19.layer.1.EncDecAttention.k.weight', 'decoder.block.7.layer.2.DenseReluDense.wi.weight', 'decoder.block.23.layer.1.EncDecAttention.o.weight', 'decoder.block.12.layer.2.DenseReluDense.wo.weight', 'decoder.block.2.layer.0.SelfAttention.q.weight', 'decoder.block.15.layer.1.EncDecAttention.v.weight', 'decoder.block.22.layer.1.EncDecAttention.o.weight', 'decoder.block.19.layer.2.layer_norm.weight', 'decoder.block.8.layer.2.layer_norm.weight', 'decoder.block.23.layer.0.layer_norm.weight', 'decoder.block.15.layer.0.SelfAttention.v.weight', 'decoder.block.8.layer.1.EncDecAttention.q.weight', 'decoder.block.10.layer.1.EncDecAttention.v.weight', 'decoder.block.20.layer.1.EncDecAttention.q.weight', 'decoder.block.4.layer.2.DenseReluDense.wo.weight', 'decoder.block.0.layer.0.SelfAttention.q.weight', 'decoder.block.22.layer.2.layer_norm.weight', 'decoder.block.1.layer.0.SelfAttention.o.weight', 'lm_head.weight', 'decoder.block.8.layer.0.layer_norm.weight', 'decoder.block.3.layer.1.layer_norm.weight', 'decoder.block.8.layer.0.SelfAttention.v.weight', 'decoder.block.23.layer.1.EncDecAttention.v.weight', 'decoder.block.17.layer.1.EncDecAttention.k.weight', 'decoder.block.11.layer.1.EncDecAttention.k.weight', 'decoder.block.8.layer.1.EncDecAttention.k.weight', 'decoder.block.2.layer.2.DenseReluDense.wi.weight', 'decoder.block.1.layer.0.SelfAttention.k.weight', 'decoder.block.18.layer.2.DenseReluDense.wo.weight', 'decoder.block.2.layer.0.SelfAttention.v.weight', 'decoder.block.3.layer.1.EncDecAttention.k.weight', 'decoder.block.21.layer.0.SelfAttention.o.weight', 'decoder.block.4.layer.1.EncDecAttention.q.weight', 'decoder.block.11.layer.1.layer_norm.weight', 'decoder.block.23.layer.0.SelfAttention.v.weight', 'decoder.block.7.layer.0.layer_norm.weight', 'decoder.block.9.layer.2.DenseReluDense.wo.weight', 'decoder.block.16.layer.2.DenseReluDense.wi.weight', 'decoder.block.7.layer.0.SelfAttention.k.weight', 'decoder.block.10.layer.2.DenseReluDense.wi.weight', 'decoder.block.16.layer.1.EncDecAttention.q.weight', 'decoder.block.17.layer.0.SelfAttention.k.weight', 'decoder.block.1.layer.1.EncDecAttention.v.weight', 'decoder.block.6.layer.1.EncDecAttention.o.weight', 'decoder.block.18.layer.0.SelfAttention.q.weight', 'decoder.block.23.layer.2.DenseReluDense.wi.weight', 'decoder.block.5.layer.0.SelfAttention.q.weight', 'decoder.block.14.layer.0.SelfAttention.v.weight', 'decoder.block.16.layer.1.EncDecAttention.o.weight', 'decoder.block.23.layer.0.SelfAttention.q.weight', 'decoder.block.6.layer.2.DenseReluDense.wo.weight', 'decoder.block.13.layer.1.EncDecAttention.q.weight', 'decoder.block.16.layer.0.SelfAttention.q.weight', 'decoder.block.1.layer.0.layer_norm.weight', 'decoder.block.9.layer.0.SelfAttention.k.weight', 'decoder.block.18.layer.0.SelfAttention.v.weight', 'decoder.block.13.layer.2.DenseReluDense.wo.weight', 'decoder.block.15.layer.2.layer_norm.weight', 'decoder.block.13.layer.2.DenseReluDense.wi.weight', 'decoder.block.14.layer.1.EncDecAttention.v.weight', 'decoder.block.17.layer.1.EncDecAttention.v.weight', 'decoder.block.7.layer.0.SelfAttention.v.weight', 'decoder.block.17.layer.1.EncDecAttention.q.weight', 'decoder.block.22.layer.0.SelfAttention.q.weight', 'decoder.block.21.layer.1.EncDecAttention.o.weight', 'decoder.block.6.layer.2.DenseReluDense.wi.weight', 'decoder.block.9.layer.1.EncDecAttention.q.weight', 'decoder.block.8.layer.1.layer_norm.weight', 'decoder.block.19.layer.1.EncDecAttention.v.weight', 'decoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight', 'decoder.block.20.layer.0.SelfAttention.q.weight', 'decoder.block.7.layer.2.DenseReluDense.wo.weight', 'decoder.block.21.layer.0.SelfAttention.v.weight', 'decoder.block.8.layer.2.DenseReluDense.wi.weight', 'decoder.block.9.layer.0.SelfAttention.v.weight', 'decoder.final_layer_norm.weight', 'decoder.block.3.layer.1.EncDecAttention.q.weight', 'decoder.block.18.layer.1.EncDecAttention.v.weight', 'decoder.block.0.layer.1.EncDecAttention.o.weight', 'decoder.block.4.layer.2.layer_norm.weight', 'decoder.block.16.layer.2.DenseReluDense.wo.weight', 'decoder.block.17.layer.2.layer_norm.weight', 'decoder.block.18.layer.1.layer_norm.weight', 'decoder.block.1.layer.1.layer_norm.weight', 'decoder.block.9.layer.1.EncDecAttention.o.weight', 'decoder.block.11.layer.0.SelfAttention.q.weight', 'decoder.block.0.layer.0.layer_norm.weight', 'decoder.block.21.layer.0.SelfAttention.k.weight', 'decoder.block.14.layer.2.layer_norm.weight', 'decoder.block.4.layer.1.EncDecAttention.v.weight', 'decoder.block.3.layer.2.DenseReluDense.wo.weight', 'decoder.block.18.layer.2.layer_norm.weight', 'decoder.block.19.layer.1.EncDecAttention.o.weight', 'decoder.block.13.layer.0.SelfAttention.q.weight', 'decoder.block.7.layer.1.EncDecAttention.o.weight', 'decoder.block.13.layer.1.EncDecAttention.k.weight', 'decoder.block.2.layer.0.layer_norm.weight', 'decoder.block.13.layer.0.SelfAttention.k.weight', 'decoder.block.15.layer.2.DenseReluDense.wo.weight', 'decoder.block.22.layer.1.EncDecAttention.k.weight', 'decoder.block.12.layer.0.SelfAttention.q.weight', 'decoder.block.16.layer.1.EncDecAttention.v.weight', 'decoder.block.12.layer.1.EncDecAttention.o.weight', 'decoder.block.22.layer.2.DenseReluDense.wi.weight', 'decoder.block.21.layer.0.layer_norm.weight', 'decoder.block.19.layer.0.layer_norm.weight', 'decoder.block.11.layer.0.SelfAttention.o.weight', 'decoder.block.20.layer.2.layer_norm.weight', 'decoder.block.2.layer.1.EncDecAttention.q.weight', 'decoder.block.10.layer.2.DenseReluDense.wo.weight', 'decoder.block.18.layer.2.DenseReluDense.wi.weight', 'decoder.block.6.layer.0.SelfAttention.v.weight', 'decoder.block.20.layer.2.DenseReluDense.wi.weight', 'decoder.block.4.layer.1.layer_norm.weight', 'decoder.block.21.layer.1.layer_norm.weight', 'decoder.block.7.layer.1.layer_norm.weight', 'decoder.block.4.layer.0.SelfAttention.q.weight', 'decoder.block.6.layer.0.layer_norm.weight', 'decoder.block.22.layer.1.EncDecAttention.v.weight', 'decoder.block.3.layer.0.SelfAttention.k.weight', 'decoder.block.22.layer.1.EncDecAttention.q.weight', 'decoder.block.13.layer.0.SelfAttention.v.weight', 'decoder.block.10.layer.0.SelfAttention.v.weight', 'decoder.block.15.layer.1.EncDecAttention.k.weight', 'decoder.block.8.layer.0.SelfAttention.q.weight', 'decoder.block.10.layer.0.layer_norm.weight', 'decoder.block.14.layer.2.DenseReluDense.wo.weight', 'decoder.block.2.layer.2.DenseReluDense.wo.weight', 'decoder.block.21.layer.2.DenseReluDense.wo.weight', 'decoder.block.8.layer.2.DenseReluDense.wo.weight', 'decoder.block.20.layer.0.SelfAttention.k.weight', 'decoder.block.2.layer.1.layer_norm.weight', 'decoder.block.5.layer.1.EncDecAttention.q.weight', 'decoder.block.12.layer.2.DenseReluDense.wi.weight', 'decoder.block.22.layer.0.SelfAttention.o.weight', 'decoder.block.17.layer.0.SelfAttention.q.weight', 'decoder.block.15.layer.0.layer_norm.weight', 'decoder.block.4.layer.0.SelfAttention.v.weight', 'decoder.block.15.layer.1.layer_norm.weight', 'decoder.block.0.layer.1.layer_norm.weight', 'decoder.block.4.layer.0.SelfAttention.o.weight', 'decoder.block.6.layer.1.EncDecAttention.v.weight', 'decoder.block.3.layer.1.EncDecAttention.v.weight', 'decoder.block.14.layer.1.EncDecAttention.k.weight', 'decoder.block.22.layer.0.SelfAttention.v.weight', 'decoder.block.13.layer.0.SelfAttention.o.weight', 'decoder.block.7.layer.2.layer_norm.weight', 'decoder.block.17.layer.0.layer_norm.weight', 'decoder.block.3.layer.0.SelfAttention.v.weight', 'decoder.block.16.layer.2.layer_norm.weight', 'decoder.block.17.layer.0.SelfAttention.v.weight', 'decoder.block.10.layer.1.EncDecAttention.q.weight', 'decoder.block.17.layer.0.SelfAttention.o.weight', 'decoder.block.12.layer.1.EncDecAttention.v.weight', 'decoder.block.5.layer.2.DenseReluDense.wi.weight', 'decoder.block.20.layer.1.EncDecAttention.o.weight', 'decoder.block.21.layer.1.EncDecAttention.q.weight', 'decoder.block.17.layer.1.layer_norm.weight', 'decoder.block.19.layer.0.SelfAttention.q.weight', 'decoder.block.18.layer.1.EncDecAttention.k.weight', 'decoder.block.16.layer.0.SelfAttention.k.weight', 'decoder.block.3.layer.0.SelfAttention.q.weight', 'decoder.block.12.layer.2.layer_norm.weight', 'decoder.block.0.layer.2.DenseReluDense.wi.weight', 'decoder.block.0.layer.1.EncDecAttention.k.weight', 'decoder.block.21.layer.0.SelfAttention.q.weight', 'decoder.block.0.layer.0.SelfAttention.k.weight', 'decoder.block.18.layer.0.layer_norm.weight', 'decoder.block.9.layer.0.layer_norm.weight', 'decoder.block.9.layer.0.SelfAttention.q.weight', 'decoder.block.11.layer.0.layer_norm.weight', 'decoder.block.14.layer.1.layer_norm.weight', 'decoder.block.5.layer.0.SelfAttention.o.weight', 'decoder.block.10.layer.1.EncDecAttention.o.weight', 'decoder.block.13.layer.1.layer_norm.weight', 'decoder.block.12.layer.1.EncDecAttention.q.weight', 'decoder.block.23.layer.2.layer_norm.weight', 'decoder.block.5.layer.1.layer_norm.weight', 'decoder.block.18.layer.0.SelfAttention.o.weight', 'decoder.block.1.layer.1.EncDecAttention.q.weight', 'decoder.block.0.layer.2.DenseReluDense.wo.weight', 'decoder.block.7.layer.1.EncDecAttention.q.weight', 'decoder.block.20.layer.0.SelfAttention.v.weight', 'decoder.block.0.layer.0.SelfAttention.v.weight', 'decoder.block.3.layer.0.layer_norm.weight', 'decoder.block.14.layer.0.SelfAttention.q.weight', 'decoder.block.16.layer.0.SelfAttention.o.weight', 'decoder.block.5.layer.1.EncDecAttention.k.weight', 'decoder.block.10.layer.1.layer_norm.weight', 'decoder.block.4.layer.1.EncDecAttention.k.weight', 'decoder.block.6.layer.1.layer_norm.weight', 'decoder.block.19.layer.0.SelfAttention.v.weight', 'decoder.block.19.layer.0.SelfAttention.o.weight', 'decoder.block.1.layer.2.DenseReluDense.wo.weight', 'decoder.block.16.layer.0.layer_norm.weight', 'decoder.block.19.layer.1.layer_norm.weight', 'decoder.block.2.layer.2.layer_norm.weight', 'decoder.block.11.layer.2.layer_norm.weight', 'decoder.block.3.layer.1.EncDecAttention.o.weight', 'decoder.block.12.layer.0.layer_norm.weight', 'decoder.block.1.layer.0.SelfAttention.v.weight', 'decoder.block.2.layer.0.SelfAttention.k.weight', 'decoder.block.21.layer.2.layer_norm.weight', 'decoder.block.5.layer.0.layer_norm.weight', 'decoder.block.8.layer.1.EncDecAttention.o.weight', 'decoder.block.20.layer.0.layer_norm.weight', 'decoder.block.5.layer.2.DenseReluDense.wo.weight', 'decoder.block.23.layer.0.SelfAttention.o.weight', 'decoder.block.9.layer.2.layer_norm.weight', 'decoder.block.14.layer.1.EncDecAttention.q.weight', 'decoder.block.0.layer.2.layer_norm.weight', 'decoder.block.15.layer.1.EncDecAttention.q.weight', 'decoder.block.15.layer.0.SelfAttention.q.weight', 'decoder.block.3.layer.2.layer_norm.weight', 'decoder.block.6.layer.1.EncDecAttention.q.weight', 'decoder.block.23.layer.1.EncDecAttention.k.weight', 'decoder.block.3.layer.0.SelfAttention.o.weight', 'decoder.block.8.layer.0.SelfAttention.k.weight', 'decoder.block.4.layer.2.DenseReluDense.wi.weight', 'decoder.block.23.layer.1.layer_norm.weight', 'decoder.block.12.layer.1.EncDecAttention.k.weight', 'decoder.block.0.layer.1.EncDecAttention.q.weight', 'decoder.block.6.layer.0.SelfAttention.q.weight', 'decoder.block.4.layer.1.EncDecAttention.o.weight', 'decoder.block.18.layer.1.EncDecAttention.q.weight', 'decoder.block.14.layer.0.layer_norm.weight', 'decoder.block.11.layer.0.SelfAttention.k.weight', 'decoder.block.10.layer.0.SelfAttention.o.weight', 'decoder.block.19.layer.2.DenseReluDense.wi.weight', 'decoder.block.22.layer.2.DenseReluDense.wo.weight', 'decoder.block.20.layer.2.DenseReluDense.wo.weight', 'decoder.block.21.layer.2.DenseReluDense.wi.weight', 'decoder.block.7.layer.0.SelfAttention.q.weight', 'decoder.block.12.layer.0.SelfAttention.v.weight', 'decoder.block.18.layer.1.EncDecAttention.o.weight', 'decoder.block.1.layer.1.EncDecAttention.o.weight', 'decoder.block.22.layer.0.SelfAttention.k.weight', 'decoder.block.11.layer.1.EncDecAttention.o.weight', 'decoder.block.2.layer.0.SelfAttention.o.weight', 'decoder.block.1.layer.1.EncDecAttention.k.weight', 'decoder.block.22.layer.1.layer_norm.weight', 'decoder.block.14.layer.0.SelfAttention.o.weight', 'decoder.block.1.layer.2.DenseReluDense.wi.weight', 'decoder.block.7.layer.1.EncDecAttention.v.weight', 'decoder.block.11.layer.1.EncDecAttention.v.weight', 'decoder.block.8.layer.1.EncDecAttention.v.weight', 'decoder.block.13.layer.2.layer_norm.weight', 'decoder.block.15.layer.1.EncDecAttention.o.weight', 'decoder.block.20.layer.0.SelfAttention.o.weight', 'decoder.block.14.layer.1.EncDecAttention.o.weight', 'decoder.block.6.layer.2.layer_norm.weight', 'decoder.block.23.layer.1.EncDecAttention.q.weight', 'decoder.block.12.layer.0.SelfAttention.k.weight', 'decoder.block.19.layer.2.DenseReluDense.wo.weight', 'decoder.block.10.layer.1.EncDecAttention.k.weight', 'decoder.block.10.layer.2.layer_norm.weight', 'decoder.block.17.layer.2.DenseReluDense.wi.weight', 'decoder.block.15.layer.0.SelfAttention.k.weight', 'decoder.block.6.layer.1.EncDecAttention.k.weight', 'decoder.block.20.layer.1.layer_norm.weight', 'decoder.block.10.layer.0.SelfAttention.q.weight', 'decoder.block.11.layer.0.SelfAttention.v.weight', 'decoder.block.2.layer.1.EncDecAttention.o.weight', 'decoder.block.7.layer.0.SelfAttention.o.weight', 'decoder.block.9.layer.2.DenseReluDense.wi.weight', 'decoder.block.12.layer.1.layer_norm.weight', 'decoder.block.20.layer.1.EncDecAttention.v.weight']\n",
      "- This IS expected if you are initializing T5EncoderModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing T5EncoderModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "All the weights of T5EncoderModel were initialized from the model checkpoint at Rostlab/prot_t5_xl_uniref50.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use T5EncoderModel for predictions without further training.\n",
      "loading file spiece.model from cache at /data2/cache/huggingface/hub/models--Rostlab--prot_t5_xl_uniref50/snapshots/973be27c52ee6474de9c945952a8008aeb2a1a73/spiece.model\n",
      "loading file added_tokens.json from cache at None\n",
      "loading file special_tokens_map.json from cache at /data2/cache/huggingface/hub/models--Rostlab--prot_t5_xl_uniref50/snapshots/973be27c52ee6474de9c945952a8008aeb2a1a73/special_tokens_map.json\n",
      "loading file tokenizer_config.json from cache at /data2/cache/huggingface/hub/models--Rostlab--prot_t5_xl_uniref50/snapshots/973be27c52ee6474de9c945952a8008aeb2a1a73/tokenizer_config.json\n",
      "loading configuration file config.json from cache at /data2/cache/huggingface/hub/models--Rostlab--prot_t5_xl_uniref50/snapshots/973be27c52ee6474de9c945952a8008aeb2a1a73/config.json\n",
      "Model config T5Config {\n",
      "  \"_name_or_path\": \"Rostlab/prot_t5_xl_uniref50\",\n",
      "  \"architectures\": [\n",
      "    \"T5ForConditionalGeneration\"\n",
      "  ],\n",
      "  \"d_ff\": 16384,\n",
      "  \"d_kv\": 128,\n",
      "  \"d_model\": 1024,\n",
      "  \"decoder_start_token_id\": 0,\n",
      "  \"dense_act_fn\": \"relu\",\n",
      "  \"dropout_rate\": 0.1,\n",
      "  \"eos_token_id\": 1,\n",
      "  \"feed_forward_proj\": \"relu\",\n",
      "  \"initializer_factor\": 1.0,\n",
      "  \"is_encoder_decoder\": true,\n",
      "  \"is_gated_act\": false,\n",
      "  \"layer_norm_epsilon\": 1e-06,\n",
      "  \"model_type\": \"t5\",\n",
      "  \"n_positions\": 512,\n",
      "  \"num_decoder_layers\": 24,\n",
      "  \"num_heads\": 32,\n",
      "  \"num_layers\": 24,\n",
      "  \"output_past\": true,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"relative_attention_max_distance\": 128,\n",
      "  \"relative_attention_num_buckets\": 32,\n",
      "  \"transformers_version\": \"4.26.1\",\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 128\n",
      "}\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ProtT5_Classfier\n",
      "Trainable Parameter: 1208142849\n",
      "ProtT5_LoRA_Classfier\n",
      "Trainable Parameter: 2508801\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tokenizer, model_reload = load_model(\"./PT5_disorder_finetuned.pth\", num_labels=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5289780",
   "metadata": {},
   "source": [
    "To check if the original and the reloaded models are identical we can compare weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e152714e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Models have identical weights\n"
     ]
    }
   ],
   "source": [
    "# Put both models to the same device\n",
    "model=model.to(\"cpu\")\n",
    "model_reload=model_reload.to(\"cpu\")\n",
    "\n",
    "# Iterate through the parameters of the two models and compare the data\n",
    "for param1, param2 in zip(model.parameters(), model_reload.parameters()):\n",
    "    if not torch.equal(param1.data, param2.data):\n",
    "        print(\"Models have different weights\")\n",
    "        break\n",
    "else:\n",
    "    print(\"Models have identical weights\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c478158",
   "metadata": {},
   "source": [
    "# Make predictions on a test set"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08533e20",
   "metadata": {},
   "source": [
    "We first load the SETH test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "5a9f505e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>sequence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18927</td>\n",
       "      <td>NREIQPPFKPKVSGKGAENFDKFFTRGQPVLTPPDQLVIANIDQSD...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>19650</td>\n",
       "      <td>LLDKDDSKAGMEEDHTYEGLDIDQTATYEDIVTLRTGEVKWSVGEHPGQ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>15141</td>\n",
       "      <td>AKRSHQAIIMSTSLRVSPSIHGYHFDTASRKKAVGNIFENTDQESL...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15768</td>\n",
       "      <td>STNPKPQRKTKRNTNRRPQDVKFPGGGQIVGGVYLLPRRGPRSQPR...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17483</td>\n",
       "      <td>DAGRKGFGEKASEALKPDSQKSYAEQGKEYITDKADKVAGKVQPED...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    name                                           sequence\n",
       "0  18927  NREIQPPFKPKVSGKGAENFDKFFTRGQPVLTPPDQLVIANIDQSD...\n",
       "1  19650  LLDKDDSKAGMEEDHTYEGLDIDQTATYEDIVTLRTGEVKWSVGEHPGQ\n",
       "2  15141  AKRSHQAIIMSTSLRVSPSIHGYHFDTASRKKAVGNIFENTDQESL...\n",
       "3  15768  STNPKPQRKTKRNTNRRPQDVKFPGGGQIVGGVYLLPRRGPRSQPR...\n",
       "4  17483  DAGRKGFGEKASEALKPDSQKSYAEQGKEYITDKADKVAGKVQPED..."
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load test sequences\n",
    "url = \"https://raw.githubusercontent.com/DagmarIlz/SETH/main/datasets/CheZOD117_test_set_sequences.fasta\"\n",
    "\n",
    "response = requests.get(url)\n",
    "response.raise_for_status()  # Check if the request was successful\n",
    "\n",
    "# Create a StringIO object to simulate a file-like object\n",
    "fasta_file = StringIO(response.text)\n",
    "\n",
    "# Load FASTA file using Biopython\n",
    "sequences = []\n",
    "for record in SeqIO.parse(fasta_file, \"fasta\"):\n",
    "    sequences.append([record.name, str(record.seq)])\n",
    "\n",
    "# Create dataframe\n",
    "df = pd.DataFrame(sequences, columns=[\"name\", \"sequence\"])\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "9f980510",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>sequence</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18927</td>\n",
       "      <td>NREIQPPFKPKVSGKGAENFDKFFTRGQPVLTPPDQLVIANIDQSD...</td>\n",
       "      <td>[0.908, -0.087, 0.389, -1.393, -0.438, -2.381,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>19650</td>\n",
       "      <td>LLDKDDSKAGMEEDHTYEGLDIDQTATYEDIVTLRTGEVKWSVGEHPGQ</td>\n",
       "      <td>[0.986, 1.055, -1.049, -0.312, 1.064, 2.683, 2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>15141</td>\n",
       "      <td>AKRSHQAIIMSTSLRVSPSIHGYHFDTASRKKAVGNIFENTDQESL...</td>\n",
       "      <td>[-0.251, 0.539, 0.032, 0.33, -2.182, -1.09, 0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15768</td>\n",
       "      <td>STNPKPQRKTKRNTNRRPQDVKFPGGGQIVGGVYLLPRRGPRSQPR...</td>\n",
       "      <td>[7.239, 8.155, 6.443, 6.22, 4.927, 4.947, 2.98...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17483</td>\n",
       "      <td>DAGRKGFGEKASEALKPDSQKSYAEQGKEYITDKADKVAGKVQPED...</td>\n",
       "      <td>[-2.544, -2.447, -1.615, -1.252, -0.973, -2.00...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    name                                           sequence  \\\n",
       "0  18927  NREIQPPFKPKVSGKGAENFDKFFTRGQPVLTPPDQLVIANIDQSD...   \n",
       "1  19650  LLDKDDSKAGMEEDHTYEGLDIDQTATYEDIVTLRTGEVKWSVGEHPGQ   \n",
       "2  15141  AKRSHQAIIMSTSLRVSPSIHGYHFDTASRKKAVGNIFENTDQESL...   \n",
       "3  15768  STNPKPQRKTKRNTNRRPQDVKFPGGGQIVGGVYLLPRRGPRSQPR...   \n",
       "4  17483  DAGRKGFGEKASEALKPDSQKSYAEQGKEYITDKADKVAGKVQPED...   \n",
       "\n",
       "                                               label  \n",
       "0  [0.908, -0.087, 0.389, -1.393, -0.438, -2.381,...  \n",
       "1  [0.986, 1.055, -1.049, -0.312, 1.064, 2.683, 2...  \n",
       "2  [-0.251, 0.539, 0.032, 0.33, -2.182, -1.09, 0....  \n",
       "3  [7.239, 8.155, 6.443, 6.22, 4.927, 4.947, 2.98...  \n",
       "4  [-2.544, -2.447, -1.615, -1.252, -0.973, -2.00...  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load test labels (CheZOD_scores) \n",
    "# here each test sequence label comes from it's own file\n",
    "\n",
    "labels=[] \n",
    "\n",
    "for n in list(df.name):\n",
    "    url = \"https://raw.githubusercontent.com/DagmarIlz/SETH/main/datasets/CheZOD117_test_set_CheZOD_scores/zscores\"+n+\".txt\"\n",
    "\n",
    "    response = requests.get(url)\n",
    "    response.raise_for_status()  # Check if the request was successful\n",
    "    \n",
    "    lines = response.text.splitlines()\n",
    "    \n",
    "    label=[]\n",
    "    \n",
    "    for l in lines:\n",
    "        label.append(l.split(\" \")[-1]) \n",
    "    \n",
    "    labels.append(label)\n",
    "\n",
    "for l in range(0,len(labels)):\n",
    "    labels[l]=[float(label) for label in labels[l]]\n",
    "    \n",
    "df[\"label\"] = labels\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "1151f4de",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_1096917/2346974570.py:4: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  test['label'] = test.apply(lambda row:  [-100 if x > 900 else x for x in row['label']], axis=1)\n",
      "/tmp/ipykernel_1096917/2346974570.py:7: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  test[\"sequence\"] = test[\"sequence\"].str.replace('|'.join([\"O\",\"B\",\"U\",\"Z\"]),\"X\",regex=True)\n",
      "/tmp/ipykernel_1096917/2346974570.py:8: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  test['sequence'] = test.apply(lambda row : \" \".join(row[\"sequence\"]), axis = 1)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sequence</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>N R E I Q P P F K P K V S G K G A E N F D K F ...</td>\n",
       "      <td>[0.908, -0.087, 0.389, -1.393, -0.438, -2.381,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>L L D K D D S K A G M E E D H T Y E G L D I D ...</td>\n",
       "      <td>[0.986, 1.055, -1.049, -0.312, 1.064, 2.683, 2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A K R S H Q A I I M S T S L R V S P S I H G Y ...</td>\n",
       "      <td>[-0.251, 0.539, 0.032, 0.33, -2.182, -1.09, 0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>S T N P K P Q R K T K R N T N R R P Q D V K F ...</td>\n",
       "      <td>[7.239, 8.155, 6.443, 6.22, 4.927, 4.947, 2.98...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>D A G R K G F G E K A S E A L K P D S Q K S Y ...</td>\n",
       "      <td>[-2.544, -2.447, -1.615, -1.252, -0.973, -2.00...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            sequence  \\\n",
       "0  N R E I Q P P F K P K V S G K G A E N F D K F ...   \n",
       "1  L L D K D D S K A G M E E D H T Y E G L D I D ...   \n",
       "2  A K R S H Q A I I M S T S L R V S P S I H G Y ...   \n",
       "3  S T N P K P Q R K T K R N T N R R P Q D V K F ...   \n",
       "4  D A G R K G F G E K A S E A L K P D S Q K S Y ...   \n",
       "\n",
       "                                               label  \n",
       "0  [0.908, -0.087, 0.389, -1.393, -0.438, -2.381,...  \n",
       "1  [0.986, 1.055, -1.049, -0.312, 1.064, 2.683, 2...  \n",
       "2  [-0.251, 0.539, 0.032, 0.33, -2.182, -1.09, 0....  \n",
       "3  [7.239, 8.155, 6.443, 6.22, 4.927, 4.947, 2.98...  \n",
       "4  [-2.544, -2.447, -1.615, -1.252, -0.973, -2.00...  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test = df[[\"sequence\", \"label\"]]\n",
    "\n",
    "# replace invalid values above 900 with -100\n",
    "test['label'] = test.apply(lambda row:  [-100 if x > 900 else x for x in row['label']], axis=1)\n",
    "\n",
    "# preprocess data\n",
    "test[\"sequence\"] = test[\"sequence\"].str.replace('|'.join([\"O\",\"B\",\"U\",\"Z\"]),\"X\",regex=True)\n",
    "test['sequence'] = test.apply(lambda row : \" \".join(row[\"sequence\"]), axis = 1)\n",
    "\n",
    "test.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9249f897",
   "metadata": {},
   "source": [
    "Then we create predictions on our test data using the model we trained before"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "c1df2299",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:10<00:00,  1.35s/it]\n"
     ]
    }
   ],
   "source": [
    "# Set the device to use\n",
    "device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n",
    "model.to(device)\n",
    "\n",
    "# Create Dataset\n",
    "test_set=create_dataset(tokenizer,list(test['sequence']),list(test['label']))\n",
    "# Make compatible with torch DataLoader\n",
    "test_set = test_set.with_format(\"torch\", device=device)\n",
    "\n",
    "# For token classification we need a data collator here to pad correctly\n",
    "data_collator = DataCollatorForTokenRegression(tokenizer) \n",
    "\n",
    "# Create a dataloader for the test dataset\n",
    "test_dataloader = DataLoader(test_set, batch_size=16, shuffle = False, collate_fn = data_collator)\n",
    "\n",
    "# Put the model in evaluation mode\n",
    "model.eval()\n",
    "\n",
    "# Make predictions on the test dataset\n",
    "predictions = []\n",
    "# We need to collect the batch[\"labels\"] as well, this allows us to filter out all positions with a -100 afterwards\n",
    "padded_labels = []\n",
    "\n",
    "with torch.no_grad():\n",
    "    for batch in tqdm(test_dataloader):\n",
    "        input_ids = batch['input_ids'].to(device)\n",
    "        attention_mask = batch['attention_mask'].to(device)\n",
    "        # Padded labels from the data collator\n",
    "        padded_labels += batch['labels'].tolist()\n",
    "        # Add batch results(logits) to predictions, we take the argmax here to get the predicted class\n",
    "        predictions += model(input_ids, attention_mask=attention_mask).logits.squeeze().tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b18401de",
   "metadata": {},
   "source": [
    "Finally, we compute our desired performance metric for the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "000028da",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Spearman:  0.721200981285582\n"
     ]
    }
   ],
   "source": [
    "# to make it easier we flatten both the label and prediction lists\n",
    "def flatten(l):\n",
    "    return [item for sublist in l for item in sublist]\n",
    "\n",
    "# flatten and convert to np array for easy slicing in the next step\n",
    "predictions = np.array(flatten(predictions))\n",
    "padded_labels = np.array(flatten(padded_labels))\n",
    "\n",
    "# Filter out all invalid (label = -100) values\n",
    "predictions = predictions[padded_labels!=-100]\n",
    "padded_labels = padded_labels[padded_labels!=-100]\n",
    "\n",
    "# Calculate classification Accuracy\n",
    "print(\"Spearman: \", stats.spearmanr(a=predictions, b=padded_labels, axis=0).correlation)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee716709",
   "metadata": {},
   "source": [
    "Great, 72.1% Spearman is a decent test performance for this test dataset (see [SETH](https://www.frontiersin.org/articles/10.3389/fbinf.2022.1019597/full) results )\n",
    "\n",
    "Further hyperparameter optimization and using a CNN prediction head will further increase performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a51c2e5a",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "provenance": [
    {
     "file_id": "https://github.com/huggingface/notebooks/blob/main/examples/protein_language_modeling-tf.ipynb",
     "timestamp": 1670229986129
    }
   ]
  },
  "gpuClass": "standard",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "023504ef79df47cd9f5672b3537d781e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_f3f0b29c16df48f798dbdabc99207f2c",
       "IPY_MODEL_fb9ce840b4644d6eab03736688e57e23",
       "IPY_MODEL_dcb559ab088446f0a4cc9ae4720f8c29"
      ],
      "layout": "IPY_MODEL_cf11d66697d84676b9d8af11a2edd064"
     }
    },
    "07e6f8ff49d74ced96b87afaf99f52c0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0a7bab101d504c7cbe0c0e21222e4010": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6e739bd554404481b4cfac9cd1710224",
      "max": 95,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_b66c522257d04820932c9c7014392136",
      "value": 95
     }
    },
    "0c9bd61e2e904fdaa4f578c19c0f6ea9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "14ba9ccca4004a1e944767648e2b1253": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d04e61e6abfe42b9afab583bcab66b92",
      "placeholder": "​",
      "style": "IPY_MODEL_279e487c47954c9abc0a57daab09f66a",
      "value": "Downloading: 100%"
     }
    },
    "14eaa8780fd543b38d5ea5da8e002b1e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9a28a789e7814ee283b8ba922ecce252",
      "placeholder": "​",
      "style": "IPY_MODEL_3149e4afc57146ceac1fdd9a1902ccbc",
      "value": "Downloading: 100%"
     }
    },
    "15e24c603abb456984509d77813a0cf7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9ef0944bc4334b32a1a5e136b3fce9e6",
      "placeholder": "​",
      "style": "IPY_MODEL_c6407f5a50b34a0d9bfbd916ffb70d17",
      "value": " 134M/134M [00:01&lt;00:00, 79.0MB/s]"
     }
    },
    "1bdf17da40fb4d8687456e5379af5da9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "1f5719ff48504d91968e1953107b060f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "279e487c47954c9abc0a57daab09f66a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "2f1f8e5d1f5343978ccf1cc5dff59718": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "305e2c8af6194f4486b505ff25be86d7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_611417b8118744c28ec2619c45b2a6ec",
      "max": 125,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_cd89abbac71740c5a013481dd9129f5b",
      "value": 125
     }
    },
    "3149e4afc57146ceac1fdd9a1902ccbc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "334e77a8b6284dfb886fefadea6998ca": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "37dce533ad7648d0a09b3ef825c5aa31": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4d579e0c846b4e1bbc360d0ff2f2eccf": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4db0117d9ac0453886964613627decc0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "51b29673275343998567b43347eb591b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "52a93c8ac4454ed1bf1a0d4711074bc2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_d154e8e8a650447ab06425706d908f4e",
       "IPY_MODEL_0a7bab101d504c7cbe0c0e21222e4010",
       "IPY_MODEL_c8b35d958ecc4f3683b86278b979f300"
      ],
      "layout": "IPY_MODEL_4db0117d9ac0453886964613627decc0"
     }
    },
    "611417b8118744c28ec2619c45b2a6ec": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6ae302430d7746edbc04b6786bdc53f5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_14ba9ccca4004a1e944767648e2b1253",
       "IPY_MODEL_e499b443a3964dc2bca05692abde2dda",
       "IPY_MODEL_15e24c603abb456984509d77813a0cf7"
      ],
      "layout": "IPY_MODEL_1f5719ff48504d91968e1953107b060f"
     }
    },
    "6c3f28dcd36d434fa8e5fe38b1d34e3c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_8d4673cb688f4dacb1e96a0d80815667",
       "IPY_MODEL_305e2c8af6194f4486b505ff25be86d7",
       "IPY_MODEL_a6f664af44464825a9c8b73215b59318"
      ],
      "layout": "IPY_MODEL_2f1f8e5d1f5343978ccf1cc5dff59718"
     }
    },
    "6e739bd554404481b4cfac9cd1710224": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "728fbf87d2ff4bed925f67c9cf0e36b7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7a00dda470ef4f0893676c794e006a8f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8921dca0901741c5ab7723ca88803543": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_14eaa8780fd543b38d5ea5da8e002b1e",
       "IPY_MODEL_abee545f24074d33a2a171f140d84ca4",
       "IPY_MODEL_d2d78bbb74664973b1cde561152e9aad"
      ],
      "layout": "IPY_MODEL_fefc4de84c7847bf88635bfcad3a5f9e"
     }
    },
    "8d4673cb688f4dacb1e96a0d80815667": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_728fbf87d2ff4bed925f67c9cf0e36b7",
      "placeholder": "​",
      "style": "IPY_MODEL_97335231c9ab4668bc8cdaee57be202f",
      "value": "Downloading: 100%"
     }
    },
    "97335231c9ab4668bc8cdaee57be202f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "9a28a789e7814ee283b8ba922ecce252": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9ef0944bc4334b32a1a5e136b3fce9e6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a6f664af44464825a9c8b73215b59318": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_fc1cf1de39dc45d69551ca2ca401a447",
      "placeholder": "​",
      "style": "IPY_MODEL_1bdf17da40fb4d8687456e5379af5da9",
      "value": " 125/125 [00:00&lt;00:00, 1.34kB/s]"
     }
    },
    "a7e27116c7b04f5e8980ea29c59bd9ce": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "a840d7d2dc0042c1955f064ef41e051d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "abee545f24074d33a2a171f140d84ca4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_07e6f8ff49d74ced96b87afaf99f52c0",
      "max": 778,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_ba9fc00d9d144278bcf944a9328d5d5f",
      "value": 778
     }
    },
    "b66c522257d04820932c9c7014392136": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "ba9fc00d9d144278bcf944a9328d5d5f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "c555968966fb402e9ed39c024476f34b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "c6407f5a50b34a0d9bfbd916ffb70d17": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c8b35d958ecc4f3683b86278b979f300": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_37dce533ad7648d0a09b3ef825c5aa31",
      "placeholder": "​",
      "style": "IPY_MODEL_e16a79b61af94733b44306e7b5444d29",
      "value": " 95.0/95.0 [00:00&lt;00:00, 3.19kB/s]"
     }
    },
    "cd89abbac71740c5a013481dd9129f5b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "cf11d66697d84676b9d8af11a2edd064": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d015260b083c40399f87d1a30be95f4f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d04e61e6abfe42b9afab583bcab66b92": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d154e8e8a650447ab06425706d908f4e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ef762a4251b6405f8065c3e4ce880688",
      "placeholder": "​",
      "style": "IPY_MODEL_4d579e0c846b4e1bbc360d0ff2f2eccf",
      "value": "Downloading: 100%"
     }
    },
    "d2d78bbb74664973b1cde561152e9aad": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_51b29673275343998567b43347eb591b",
      "placeholder": "​",
      "style": "IPY_MODEL_334e77a8b6284dfb886fefadea6998ca",
      "value": " 778/778 [00:00&lt;00:00, 26.3kB/s]"
     }
    },
    "dcb559ab088446f0a4cc9ae4720f8c29": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_eb8017bf12034a0297f17f0f3f5f1874",
      "placeholder": "​",
      "style": "IPY_MODEL_a840d7d2dc0042c1955f064ef41e051d",
      "value": " 93.0/93.0 [00:00&lt;00:00, 3.23kB/s]"
     }
    },
    "e16a79b61af94733b44306e7b5444d29": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e499b443a3964dc2bca05692abde2dda": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ebb2f370c71a470895096be1ab1eadbf",
      "max": 134360208,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_a7e27116c7b04f5e8980ea29c59bd9ce",
      "value": 134360208
     }
    },
    "eb8017bf12034a0297f17f0f3f5f1874": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ebb2f370c71a470895096be1ab1eadbf": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ef762a4251b6405f8065c3e4ce880688": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f3f0b29c16df48f798dbdabc99207f2c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7a00dda470ef4f0893676c794e006a8f",
      "placeholder": "​",
      "style": "IPY_MODEL_0c9bd61e2e904fdaa4f578c19c0f6ea9",
      "value": "Downloading: 100%"
     }
    },
    "fb9ce840b4644d6eab03736688e57e23": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d015260b083c40399f87d1a30be95f4f",
      "max": 93,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_c555968966fb402e9ed39c024476f34b",
      "value": 93
     }
    },
    "fc1cf1de39dc45d69551ca2ca401a447": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fefc4de84c7847bf88635bfcad3a5f9e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
